Overview
Research introduces SportD, a dataset and evaluation framework designed to assess the capacity of vision-language models (VLMs) for physical strategic decision-making. The study utilizes soccer as an objective testbed, providing scenarios with quantifiable action values. SportD comprises 1415 decision scenarios extracted from professional men's and women's soccer games. In these scenarios, a VLM observes the moments preceding a decision and is tasked with selecting the subsequent action.
Research Context
Traditional evaluations of vision-language models often focus on their ability to describe visual scenes. This research extends the inquiry to examine whether these models can perform effectively within a scene by making sound strategic decisions. The chosen domain of soccer offers a controlled environment where actions possess objectively quantifiable values, facilitating rigorous analysis of strategic choices.
Approach
The methodology centers on the SportD dataset, which contains 1415 decision scenarios. For each scenario, VLMs were presented with visual information representing the seconds before a critical decision point. Their task was to choose the optimal next action from a set of possibilities. The evaluation then quantified the models' performance based on the optimality of their chosen actions.
Findings
- Optimal Action Selection: Vision-language models selected the optimal action in approximately 30% of the evaluated scenarios. This performance level is noted to be less frequent than human decision-making in similar contexts.
- Action Preference and Conservatism: VLMs demonstrated a clear preference for actions characterized as 'safer.' These safer actions were identified as having lower variance, lower value, and contributing less physical progress toward the goal.
- Success Probability Estimation: Frontier VLMs exhibited proficiency in estimating the success probability of actions. They placed the action with the highest success probability among their top choices in 83% to 92% of cases.
- Conflation of Likelihood and Value: A systematic bias was observed wherein VLMs conflated the likelihood of an action's success with its perceived value. They assigned higher value to actions that were more likely to succeed. This relationship was quantified with Spearman correlations ranging from $0.30$ to $0.52$.
- Absence of Ground Truth Correlation: The observed conflation by VLMs contrasts with the ground truth data, where no such relationship exists between the likelihood of success and the value of an action. The ground truth exhibited a Spearman correlation of $-0.08$.
- Mechanism of Conservatism: The conservatism displayed by VLMs, characterized by a preference for safer actions, is attributed to a mis-calibration of value, stemming from the conflation of likelihood and value.
Why This Matters
SportD establishes a novel direction for the rigorous evaluation of physical strategic decision-making capabilities in VLMs. It provides a framework for decomposing model choices to reveal underlying mechanisms, such as systematic biases like risk aversion. This work contributes to understanding the strategic limitations of current VLM architectures beyond scene description.