Readout is not Recovery: Dissociating Coordinate Emission from Visual-Corruption Repair in Vision-Language Models

Vision-language models (VLMs) ground a referring expression in one of two ways. They decode the box as coordinate tokens, which is precise but takes one sequential step per token, or they regress it from a hidden state with a small head, which takes one step but loses accuracy at high IoU. The two fail differently: a head usually finds the right object but places its edges loosely, while coordinate tokens place edges precisely. We therefore use the regression head as a speculative drafter for the model's own coordinate tokens. Locus writes the head's box as coordinate tokens and checks them with the same backbone by Jacobi iteration over a shared image prefix. The fixed point is the model's greedy coordinate-token answer, reached in about 5.60 sequential passes on average instead of 18.46 decoding steps; head and verifier are one set of weights, trained jointly. On the 8 RefCOCO, RefCOCO+ and RefCOCOg splits, verification raises Locus-2B's [email protected] from 54.87 for its own head to 59.13, and the one-pass head is ahead at [email protected] of every published model with at most 3B parameters except Florence-2-L. Against Jag, the strongest open regression-head model, at its own size and training budget, Locus ties at [email protected] and, with typical acceptance, gains +4.52 points at [email protected], with paired significance tests. Exact answers come 1.5x faster than vLLM greedy-decoding the same coordinate tokens with the same weights; a pure regression head remains faster still.

Generative Vision-Language Models (VLMs) express bounding boxes as coordinate sequences, whose token identities often leave axis and place value implicit. We study how explicitly encoding these roles improves visual grounding. Hi-Token assigns a distinct token to each combination of axis, decimal place, and digit value, representing a box with 12 tokens drawn from 60 types. This design exposes coarse-to-fine coordinate structure and increases token reuse relative to atomic location tokens while retaining the existing VLM architecture. Hi-GAR complements this representation with a geometry-based reward for Group Relative Policy Optimization (GRPO), combining box overlap, coordinate accuracy at multiple scales, strict-IoU bonuses, and validity gating. Matched training comparisons show that Hi-Token improves localization throughout the evaluated IoU range, with gains across three VLM backbones. Hi-Token also outperforms an additionally tuned atomic-token baseline on the RefCOCO family before reinforcement learning. Hi-GAR further improves localization and is used only during training. The resulting Hi-R1 model achieves higher values than strong specialist baselines on most reported metrics. Token-frequency and geometric diagnostics characterize the representation and reward stages. These results establish explicit axis and place-value encoding as an effective design for generative visual grounding. Project page: https://xyzzzh.github.io/Hi-Token/

Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is treated as neutral. We find it is not, and that the limits a benchmark reports can belong to the readout rather than to the model. On 200 COCO photographs, Qwen3-VL-4B picks the correct one of nine locations for a named object 68.5% of the time when the locations are given in English and 20.0% when the same locations are given as pixel coordinates. Chance is 11.1%. The cost arises when the answer options are coordinates; giving the model a coordinate in the question instead costs 3.5 points and is not significant. The gap holds on a 4x4 grid, under 8-bit rather than 4-bit quantization, and in every slice by object size, boundary distance and category. It also decides which model wins. Two models that tie under English names differ by 39 points in one coordinate system and by 54 in the other, in opposite directions. On the color task, three of the four open models capable of the task show the penalty; on photographs, two of three open models do, and so does Gemini, at 11.1 points on parseable answers (p = 1e-4). GPT-4o does not. To ask whether a model reads a coordinate at all, we attach the wrong name to each one and record which the model follows. Color options written as hue angles are followed below chance; a normalized pixel convention is followed at four times chance. This tells apart conventions a model can use from ones it cannot, though it did not predict accuracy on two untried conventions. Five models also name the same color wheel five different ways, so a fixed answer vocabulary is not neutral across models either. Five times during this work we measured a capable model as incapable because our scorer and the model disagreed about what an answer looks like. We report each case. They are the phenomenon in miniature.