Overview
Generative Vision-Language Models (VLMs) often represent bounding-box coordinates as independent output symbols, a method which does not explicitly encode numerical order or axis semantics. This representation has been identified as a source of error in visual grounding tasks. To address this, a hierarchical coordinate tokenization scheme, named Hi-Token, has been developed. Hi-Token encodes each coordinate using axis-specific tokens for the hundreds, tens, and ones digits. This approach introduces a coarse-to-fine structure and enhances token reuse while maintaining compatibility with existing VLM architectures. Complementing Hi-Token is Hi-GAR, a geometry-based reward mechanism designed for Group Relative Policy Optimization (GRPO). Hi-GAR leverages box overlap and coordinate accuracy across multiple scales and is applied during the training phase only.
Approach
The core of the approach involves two primary components: Hi-Token and Hi-GAR. Hi-Token modifies the representation of bounding-box coordinates. Instead of treating coordinates as single, independent symbols, Hi-Token assigns distinct tokens based on the digit's place value (hundreds, tens, ones) and the coordinate's axis. This method aims to embed structural information about the coordinates into the token representation. The explicit encoding of digit boundaries and axis specificity contributes to the coarse-to-fine granularity of the coordinate representation.
Hi-GAR operates as a training-specific component. It introduces a geometry-based reward function into the Group Relative Policy Optimization (GRPO) framework. This reward function is constructed from measures of box overlap and coordinate accuracy, evaluated at different scales. The application of Hi-GAR is restricted to the training phase to guide the model's learning process towards more accurate localization.
The combined methodology was evaluated through controlled comparisons. These comparisons were conducted under matched training conditions to isolate the effects of Hi-Token and Hi-GAR. Experiments were performed on three different VLM backbones and across the RefCOCO family of benchmarks. Analyses also investigated token frequency, the influence of digit boundaries, object scale variations, and IoU distributions to further elucidate the mechanisms through which the coordinate representation and reward training impact performance.
Findings
- Hi-Token, when compared under matched training conditions, demonstrated improvements in localization across the entire evaluated Intersection over Union (IoU) range.
- Hi-GAR further contributed to performance by reducing the incidence of low-overlap predictions. Hi-GAR's benefits were observed despite its exclusive use during the training phase.
- Experiments conducted across three distinct VLM backbones and the RefCOCO family of benchmarks indicated consistent performance gains. These gains were observed across different models and benchmark datasets.
- The Hi-R1 configuration, which incorporates these advancements, achieved higher metric values compared to strong specialist baselines for most reported metrics.
- Analyses of various factors, including token frequency, digit boundaries, object scale, and IoU distributions, provided explanations for the observed effects attributed to the coordinate representation and the reward-based training.
- The results collectively suggest that the generation of structured coordinates offers an effective method for improving generative visual grounding.
Why This Matters
The identified representation of bounding-box coordinates as independent symbols constitutes a significant source of error in visual grounding within generative VLMs. Addressing this through structured coordinate generation, as achieved by Hi-Token and Hi-GAR, offers a method to enhance the localization capabilities of these models.