Overview
Research introduces X-Stream, a benchmark designed for multi-stream streaming understanding. This benchmark addresses a perceived gap in current evaluation methodologies, which are largely confined to single-stream paradigms, by focusing on continuous, multi-stream interactions inherent in real-world applications. The study also explores the performance of multi-modal large language models (MLLMs) when conceptualized as multiplexers under the lens of Signal Multiplexing Theory.
Research Context
While video streaming understanding has progressed, certain applications like live sports broadcasting, autonomous driving, and multi-screen collaboration necessitate multi-stream interactions. Existing benchmarks primarily evaluate single-stream understanding, leaving a gap for the assessment of online, cross-stream reasoning capabilities. The research recognizes the need for an evaluation framework that reflects these multi-stream demands.
Approach
The development of X-Stream involved the curation of 4,220 rigorously verified question-answer (QA) pairs across 932 videos. This dataset is structured to evaluate 11 distinct subtasks that span multi-window, multi-view, and multi-device scenarios. A dual-verification pipeline was employed during dataset construction to mitigate potential over-reliance on a single stream. The research also pioneers a conceptual framework where MLLMs are viewed as naive multiplexers. Their performance was systematically evaluated using online inference experiments guided by Signal Multiplexing Theory.
Findings
- X-Stream is presented as the first benchmark specifically dedicated to multi-stream streaming understanding.
- The benchmark comprises 4,220 QA pairs derived from 932 videos.
- It evaluates 11 subtasks covering multi-window, multi-view, and multi-device contexts.
- The dataset's construction utilized a dual-verification pipeline, designed to prevent over-reliance on a single stream.
- Online inference experiments revealed that state-of-the-art MLLMs struggle significantly with concurrent streams.
- These MLLMs achieved approximately 50% scores in the evaluations.
- The MLLMs demonstrated poor proactive ability in handling multi-stream scenarios.
- The findings expose trade-offs associated with current multiplexing schemes.
Why This Matters
The X-Stream benchmark provides an evaluation protocol for multi-stream agents, addressing a gap in previous single-stream focused assessments. The empirical guidance derived from evaluating MLLMs as multiplexers offers insights for the development of future multi-stream systems, particularly in applications requiring simultaneous processing of multiple video feeds.