Overview
Multimodal instruction tuning, a process involving training models with combined image and video data, frequently encounters computational inefficiencies. This inefficiency stems from the practice of distributing computational resources across large, heterogeneous image-video datasets where the utility of individual data points is highly variable. To address this, a framework termed Goal-Driven Data Optimization (GDO) has been developed.
GDO operates by computing six distinct sample descriptors for each candidate data sample. These descriptors are then used to construct optimized training subsets, tailored for specific training objectives. The framework's efficacy was evaluated under a fixed training protocol: a one-epoch Qwen3-VL-8B-Instruct training, conducted on eight H20 GPUs.
Research Context
The core challenge identified in multimodal instruction tuning is the compute-intensive nature of current approaches. This is attributed to the inclusion of large, mixed image-video data pools during training, where the individual contributions or relevance of samples within these pools are often inconsistent. Such data distribution leads to computational resources being allocated inefficiently across samples that may not all contribute equally to the learning process or to specific training goals. The implicit context is the drive to improve the efficiency and performance of large multimodal models.
Approach
The GDO framework employs a data optimization strategy. It begins by generating six specific sample descriptors for each data sample considered for training. These descriptors serve as a basis for evaluating the characteristics and potential utility of each sample. Subsequently, GDO utilizes these descriptors to assemble optimized training subsets. The selection of samples for these subsets is goal-driven, meaning the subsets are designed to align with particular training objectives. The framework's operational methodology aims to concentrate training efforts on data deemed more pertinent to achieving defined goals, thereby reducing the overall volume of data processed without compromising, and potentially enhancing, performance.
Findings
Under the specified training and evaluation protocol, GDO demonstrated several performance improvements compared to a fixed 512k-sample Uni-10x baseline:
- Reduced Sample Count for Convergence: GDO achieved the performance reference of the Uni-10x baseline using significantly fewer training samples across multiple benchmarks:
- 35.4k samples on MVBench.
- 26.6k samples on VideoMME.
- 27.3k samples on MLVU.
- 34.7k samples on LVBench.
- Improved Accuracy: Beyond faster convergence, GDO also resulted in higher accuracy metrics:
- +1.38 percentage points on MVBench.
- +1.67 percentage points on VideoMME.
- +3.08 percentage points on MLVU.
- +0.84 percentage points on LVBench.
- Benchmark-Specific Gains: The most substantial gains in accuracy were observed on MVBench and MLVU. Gains on LVBench were more modest, which is consistent with its ultra-long-video setting and the prevailing short-video/image-dominant nature of the training data pool used.
- Temporal Emphasis Impact: Across different configurations (MinLoss, Diverse, Temp, and Temp+), a stronger emphasis on temporal features within the GDO framework led to progressively improved understanding of long-video content.
In summation, GDO enabled faster convergence with a reduced number of training samples, while simultaneously achieving superior accuracy under a fixed training protocol, compared to the Uni-10x baseline.
Potential Applications
The source material explicitly states:
Code is available at https://github.com/rujiewu/GDO.