Vision-language models (VLMs) increasingly rely on point coordinates as a compact and executable interface for visual grounding in GUI interaction, robotic manipulation, and interactive visual systems. However, learning reliable pointing behavior remains difficult because the supervision space is inherently non-unique: many coordinates may be valid within the same target region, while multi-instance instructions require target coverage, count consistency, and duplicate suppression. This work presents PointRL, a verifiable reinforcement learning framework that learns point-level grounding from existing heterogeneous annotation evidence. PointRL converts bounding boxes, masks, and instance labels into pointing instructions, while retaining their target supports, instance membership, and set constraints as hidden verifier evidence, i.e., annotations kept outside the prompt and used by a deterministic checker to score predictions. The proposed reward evaluates parseability, point validity, instance coverage, cardinality consistency, and redundant or missing predictions. On PointArena, PointRL improves the overall accuracy of Qwen3.5-4B from 56.11% to 65.58%. Further evaluations on RoboSpatial, BLINK, and Ref-Adv show same-backbone gains on the evaluated external benchmarks, suggesting that verifiable point-level feedback may benefit spatial grounding in these settings.
Figures & tables
Fig. 1: Task illustration for verifiable point-level grounding. A valid response should place points on the intended target supports, while nearby distractors, incomplete coverage, or shortcut outputs are rejected by the verifier.
Fig. 2: Construction of verifiable pointing samples from raw annotations. The pipeline normalizes annotation sources into object entities, collects supporting evidence, and generates the instruction with hidden verifier evidence V=(T,C) . The target set and constraints are then fixed by rule-based checks.
Rule
Count
Evidence
Check
Functional
262
AGD regions
Affordance support
Instance
345
COCO/PixMo instances
Coverage/count
Semantic
249
Class labels
Category match
Relation
393
Entity pairs
Spatial relation
Cue
398
Cue-target pairs
Shortcut guard
Total
1,647
Mixed evidence
Rule-checked targets
TABLE I: Composition of accepted PointRL-Data rules.
Fig. 3: Point reward computation from hidden verifier evidence. The reward module parses the final answer into a point collection, matches predictions to hidden verifier targets with Hungarian matching, and combines matched-pair localization quality with set-level coverage and count checks. Format and instruction-conditioned guards suppress invalid or shortcut responses before returning the core verifier point reward.
Fig. 4: Local and global reward behavior for a multi-instance query. Green circles lie in target supports and red crosses lie outside. Local reward measures point-target quality, whereas global reward penalizes missed targets, incorrect counts, and collapsed predictions. The same formulation handles singleton targets.
Fig. 5: Qualitative comparison on PointArena Point-Bench. Each category panel shows Qwen3.5-4B on the left and Qwen3.5-4B + PointRL on the right. Red and green circular markers denote the corresponding baseline and PointRL predictions; blue rings mark reference or current points in steerable examples, and translucent green regions indicate target supports when shown.
Method
Params
Affordance
Counting
Reasoning
Spatial
Steerable
Overall Acc
GPT-4o
N/D
42.42
31.12
23.83
25.64
24.50
29.50
Qwen2.5-VL-7B-Instruct
7B
76.77
52.55
53.89
61.54
36.50
56.25
Qwen2.5-VL-72B-Instruct
72B
76.77
57.14
54.40
60.00
46.50
58.96
Molmo-72B
72B
87.88
54.59
69.43
70.26
37.00
63.83
Gemini-Robotics-ER-1.5
N/D
69.70
68.47
60.10
69.74
67.50
67.10
Qwen3.5-2B
2B
34.34
10.20
32.12
33.85
16.00
25.25
TABLE II: Main results on PointArena Point-Bench.
Fig. 6: Comparison of reward configurations on multi-target pointing examples. Each row is one query; columns show the pretrained baseline, localization-only reward, localization plus target coverage, cue-copying shortcut cap, cue-copying shortcut cap with progress shaping, and the full PointRL reward. Red points denote predicted point collections.
Method
Affordance
Counting
Reasoning
Spatial
Steerable
Overall Acc
Qwen3.5-4B
65.66
47.45
58.55
69.23
40.00
56.11
+ Loc. only
69.19
59.18
60.62
69.23
43.00
60.18
+ Loc. + Cov.
67.17
57.14
68.91
71.28
48.00
62.42
+ Cue cap
67.68
55.10
66.32
68.72
49.50
61.41
+ Cue cap + Prog.
69.70
56.63
66.84
66.15
41.00
59.98
+ Full reward
75.25
60.71
70.47
71.28
50.50
65.58
TABLE III: Reward ablation on PointArena Point-Bench.
Fig. 7: External-benchmark examples on Ref-Adv and RoboSpatial-Home. In each pair, the left panel shows Qwen3.5-4B and the right panel shows Qwen3.5-4B + PointRL. In the Ref-Adv row, red boxes are predictions, blue boxes are targets, and IoU is reported above each panel. In the RoboSpatial-Home row, green regions mark valid target regions; red and green dots denote baseline and PointRL predictions, respectively.
Fig. 8: Ref-Adv predictions under increasing distractor counts. The three example groups correspond to 2–3, 4–6, and ≥7 distractors from left to right; within each group, the left panel is Qwen3.5-4B and the right panel is Qwen3.5-4B + PointRL. Red boxes denote predictions, blue boxes denote targets, and IoU is reported above each panel.
Method
Context
Config.
Compat.
BLINK
Molmo
0.00
48.70
24.80
67.10
GPT-4o
6.60
74.00
55.20
76.20
RoboPoint
2.50
59.40
80.10
63.60
RoboPoint + RoboSpatial +
21.30
70.00
88.60
70.60
LLaVA-NeXT + RoboSpatial +
19.70
76.40
80.10
79.00
Qwen3.5-4B (baseline)
27.05
79.67
25.71
66.43
TABLE IV: External evaluation on RoboSpatial and BLINK. All values are percentages under the official benchmark metrics. The three RoboSpatial columns report the Context, Configuration, and Compatibility splits. Methods marked with + use benchmark-specific training data.