Overview
Research introduces SUTURE, a novel verifier designed for video temporal grounding within the framework of reinforcement learning with verifiable rewards (RLVR). This system utilizes the joint structure of rollout groups, departing from conventional approaches that typically score each rollout independently. SUTURE integrates disagreement across rollouts to control target reweighting and employs coverage at each position to redistribute reward mass, exploiting structure at two complementary scales.
Research Context
Video temporal grounding involves identifying specific temporal intervals within a video. Reinforcement learning with verifiable rewards (RLVR) provides a framework for adapting pretrained models to this task, where generated temporal intervals can be directly scored against ground truth intervals. However, existing overlap verifiers commonly score individual rollouts without leveraging the joint structure inherent in a group of rollouts.
Approach
SUTURE conditions its verification process on the entire rollout group. Its methodology exploits this structure at two distinct, yet complementary, scales:
- Disagreement Across Rollouts: This mechanism dictates the strength of target reweighting.
- Coverage at Each Position: This aspect determines how reward mass is redistributed within the group.
The resulting SUTURE verifier allows for an exact decomposition. This decomposition separates the verifier into the standard Intersection over Union (IoU) term and a covariance correction component, which is determined by the characteristics of the rollout group. A local gradient diagnostic was employed to analyze group responses. This diagnostic identified a preference for responses that cover relatively less supported target regions within the analyzed groups.
Findings
- SUTURE improves grounding performance across all reported IoU thresholds on five distinct temporal grounding benchmarks.
- The trained policy, when utilizing SUTURE, exhibits reduced video-start anchoring in reasoning traces. Specifically, for later events, the first temporal mention more frequently overlaps the annotated target.
- The joint structure of a rollout group can support a more informative temporal verifier.
- The SUTURE verifier admits an exact decomposition into the standard IoU term and a covariance correction, which is determined by the rollout group.
Why This Matters
The findings indicate that by leveraging the joint structure of rollout groups, video temporal grounding can achieve improved performance and more accurate reasoning traces, particularly in identifying events later in a video. This suggests a more effective method for verifying temporal intervals in complex video analysis tasks.