BiGrounder: Bidirectional Training Unlocks the Potential of GUI Grounding
Authors: Dengyao Shi, Tianyu Fu, Fei Hu, Anyang Su, Hanning Wang, Shuo Wang, Minghui Wu, Chenxu Zhao, Zhen Lei
Organizations: MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Mininglamp Technology
Graphical User Interface (GUI) grounding aims to map natural language instructions to precise operational coordinates, serving as a critical task for autonomous agent interaction. Current approaches formalize GUI grounding as predicting action-coordinate pairs from instructions, with end-to-end training supervised solely by final coordinates. Training datasets typically provide only a small number of descriptions per target coordinate, whereas real-world scenarios involve a much larger set of varied descriptions for the same target, which limits robustness to linguistic variation. While recent GUI grounding paradigms have begun employing reinforcement learning to enhance localization accuracy, models with limited exploratory capability before reinforcement yield constrained improvements even with identical training data during the reinforcement phase. To overcome these limitations, we propose a bidirectional reward paradigm that enables model self-reinforcement. During SFT, bidirectional training equips the model to generate action descriptions from target coordinates. During RL, the model samples semantically consistent instruction variations on-policy and grounds each variation back to its target. A cycle-consistency gate couples semantic quality to grounding success, turning the model's own instruction policy into a source of continually refreshed reinforcement trajectories. Our 4B-parameter model achieves competitive accuracy across five grounding benchmarks in our evaluation, indicating the effectiveness of the proposed training paradigm. https://github.com/Spell-shi/BiGrounder
Figures & tables
Figure 1: Existing datasets (a) suffer from limited instruction coverage, whereas our bidirectional reinforcement framework (b) samples on-policy instruction variations for the same UI element, strengthening semantic–element understanding.
Figure 2: During the SFT stage, we organize the training data into six tasks: instruction generation, instruction grounding, bounding-box description generation, bounding-box localization, point-based description generation, and point localization.
Figure 3: The reinforcement stage couples on-policy instruction-generation trajectories with their corresponding instruction-grounding trajectories.
Model
ScreenSpot-Pro
ScreenSpot-V2
MMBench-GUI L2
UI-Vision
OSWorld-G-R
Avg.
Qwen2.5-VL-3B backbone
Qwen2.5-VL-3B ∗ ( Bai et al. 2025b )
25.9
80.8
52.0
0.7
39.4
39.8
UI-R1-E-3B ( Lu et al. 2025 )
17.8
88.6
68.4
16.5
48.8
48.0
SE-GUI-3B ( Yuan et al. 2025 )
35.9
86.8
66.3
15.0
46.1
50.0
InfiGUI-R1-3B ( Liu et al. 2025b )
35.7
89.5
70.6
17.8
42.9
51.3
GUI-G 2 -3B ( Tang et al. 2025 )
36.4
87.6
66.3
18.7
53.5
52.5
Table 1: Grounding accuracy on five GUI grounding benchmarks and their average, grouped by backbone. Unless marked with “ ∗ ”, baseline results are taken from the corresponding original papers or technical reports (cited in each row), or from the consolidated evaluation reported by GroundNext ( Feizi et al. 2025 ) . “ ∗ ” marks results we re-evaluated under our unified protocol for benchmarks not reported in these sources. “Avg.” is the mean over the five benchmarks. Within each backbone group, best is in bold and second-best is underlined. The green subscript denotes the absolute Avg. gain over each model’s backbone.
Executor Model
AndroidWorld
OSWorld
Aria-UI ( Yang et al. 2024 )
44.8
15.2
UGround-V1-7B ( Gou et al. 2025 )
32.8
16.1
UI-TARS-1.5-7B ( Seed 2025 )
37.9
19.1
ScaleCUA-7B ( Liu et al. 2025c )
48.3
28.1
Qwen3-VL-4B ∗ ( Bai et al. 2025a )
68.9 ± 0.9
40.2 ± 0.8
BiGrounder-4B
72.4 ± 0.9
42.8 ± 0.5
Table 2: GUI agent benchmark results with each model as the grounding executor and GPT-4o as the planner. AndroidWorld follows predefined steps; OSWorld uses a 50-step budget. Best in bold. ∗ Evaluated by us (mean ± std over 3 runs).
Method
Original Accuracy ↑
Any Correct ↑
All Correct ↑
Any–All Gap ↓
Qwen3-VL-4B
65.3
73.4
56.0
17.4
Bi-SFT
65.7
73.6
56.8
16.8
Bi-SFT + Forward RL
67.4
74.5
58.2
16.3
BiGrounder-4B
69.5
75.3
60.5
14.8 ↓2.6
Table 3: Robustness to instruction variation on OSWorld-G-R-Variants . Each original instruction is paired with three target-preserving variants that differ in wording and sentence structure. Original : accuracy under the original instruction; Any-/All-Correct : element grounded correctly under at least one/all four instructions. Any–All Gap : sensitivity to instruction variation. The green subscript denotes the absolute gap reduction over Qwen3-VL-4B. Best in bold.
Figure 4: RL training dynamics. (a) semantic consistency (BERTScore) and within-group diversity ( 1− Self-BLEU) of the on-policy instructions. (b) cycle-grounding reliability measured by Any@4 and All@4, with the shaded Any–All gap narrowing over training.
Figure 5: Qualitative comparison of grounding under instruction variations.
Method
SS-Pro
SS-V2
OS-G-R
UI-V.
MM-L2
Avg.
Qwen3-VL-4B
59.3
93.5
65.3
26.9
81.2
65.2
Single-SFT
59.4
90.6
65.2
39.7
79.8
66.9 +1.7
Bi-SFT
60.2
91.2
65.7
39.9
81.0
67.6 +2.4
+RL-InBbox
63.2
92.9
67.4
44.9
84.1
70.5 +5.3
+RL-Distance
62.7
92.3
66.8
42.5
83.5
69.6 +4.4
+RL-Gaussian
63.5
92.9
67.9
44.8
85.1
70.8 +5.6
Table 4: Controlled comparison of training recipes with Qwen3-VL-4B-Instruct backbone and complete training dataset. The subscript in Avg. denotes the absolute gain over the base model.
Method
SS-Pro
SS-V2
OS-G-R
UI-V.
MM-L2
Avg.
Bi-SFT
60.2
91.2
65.7
39.9
81.0
67.6
RL w/o Cycle-C.
62.8
92.4
66.9
44.9
83.5
70.1 +2.5
RL w/ Cycle-C.
66.6
93.9
69.5
46.6
87.5
72.8 +5.2
Table 5: Ablation on cycle-consistency gating for mitigating reward hacking. The subscript in Avg. denotes the absolute gain over the Bi-SFT baseline.
Method
SS-Pro
SS-V2
OS-G-R
UI-V.
MM-L2
Avg.
Qwen3-VL-4B
59.3
93.5
65.3
26.9
81.2
65.2
Bi-SFT w/o rewritten
59.7
91.2
65.5
39.5
80.6
67.3
Bi-SFT w/ rewritten
60.2
91.2
65.7
39.9
81.0
67.6
BiGrounder w/o rewritten
64.7
93.1
68.1
44.8
85.6
71.3
BiGrounder w/ rewritten
66.6
93.9
69.5
46.6
87.5
72.8
Table 6: Effect of Gemini-rewritten instructions under the same backbone and training recipe.
Figure 6: Training dynamics under different reward hyperparameters: (a) semantic-reward threshold t ; (b) pixel-level tolerance τ .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Instruction following comparison before and after cycle-consistency reinforcement, showing improved spatial grounding.
Figure A.2: Instruction following comparison before and after cycle-consistency reinforcement, showing improved icon grounding.
Figure A.3: Analyzing the semantic generation capability before and after reinforcement learning.
Figure B.1: Composition of the 4,482k-pair training corpus. The inner ring splits the data into forward (instruction → coordinate) and reverse (coordinate → instruction) halves of equal size; the outer ring breaks each half down by data source and grounding modality. The GroundCUA portion uses Video-Audio Production samples and their reverse counterparts. As an optional augmentation, the 790k Grounding bbox samples—those that already carry bounding-box annotations—have their instructions rewritten with Gemini-2.5-Flash to broaden localization-based reward coverage during RL; the core pipeline can be trained on open-source data alone.
Filter
Kept (C / I)
Precision ↑
NFR ↑
No filter
273 / 27
91.0% (273/300)
0.0% (0/27)
BERTScore >t
268 / 22
92.4% (268/290)
18.5% (5/27)
LLM Judge
267 / 7
97.4% (267/274)
74.1% (20/27)
Cycle-Consistency
260 / 2
99.2% (260/262)
92.6% (25/27)
Appendix
Table D.1: Noise filtering on 300 manually annotated reverse instructions, including 273 correct and 27 incorrect ones. Precision : fraction of correct samples in the kept set. Noise Filtering Rate (NFR) : percentage of incorrect samples successfully filtered out.
Reverse Instruction
BERTScore
Cycle-C.
Reward
Click the star to follow the repository.
0.86
✗
0.00
Click the star to bookmark this webpage.
0.92
✓
0.92
Appendix
Table D.2: Case study showing that Cycle-Consistency filters out high-BERTScore but visually incorrect reverse instructions. GT: “Click star to save this page.”
Method
Rev.
Fwd.
SS-Pro
SS-V2
OS-G-R
MM-L2
Avg.
Forward-only RL ( 1×16 )
0
16
63.7
92.9
67.4
85.3
77.3
Forward-only RL ( 1×20 )
0
20
63.9
92.9
67.9
85.1
77.5
BiGrounder Gate-only
4
16
65.6
92.9
68.6
86.2
78.3
BiGrounder
4
16
66.6
93.9
69.5
87.5
79.4
Appendix
Table E.1: Rollout-budget and reverse-reward controls. Rev./Fwd.: reverse/forward trajectories per target. SS-Pro: ScreenSpot-Pro; SS-V2: ScreenSpot-V2; OS-G-R: OSWorld-G-Refine; MM-L2: MMBench-GUI L2. Scores are accuracy (%). Avg. is the unweighted mean over these four benchmarks, excluding UI-Vision. The 1×20 baseline matches BiGrounder’s total trajectory count. Gate-only uses binary reverse rewards.
Method
Original
Any ↑
All ↑
Gap ↓
Forward-only RL
67.4
74.5
58.2
16.3
Offline aug.
68.2
75.0
59.1
15.9
Online aug.
68.8
75.2
60.1
15.1
BiGrounder
69.5
75.3
60.5
14.8
Appendix
Table E.2: Instruction-generation and reverse-update ablations on OSWorld-G-Refine-Variants. Original, Any, and All are accuracies (%); Gap is Any minus All in percentage points. Offline/online augmentation use a frozen Bi-SFT/current-policy instruction generator, respectively, with forward-only updates; BiGrounder also updates the reverse policy. These three methods share rollout budgets and optimization-step counts.
Method
SS-Pro
SS-V2
OS-G-R
MM-L2
Avg.
Qwen3.5-9B
65.2
91.8
67.7
80.4
76.3
+ BiGrounder
65.7
92.0
68.0
85.1
77.7
Qwen3.5-35B-A3B
64.5
95.2
63.6
87.5
77.7
+ BiGrounder
65.3
94.7
67.6
88.1
78.9
Appendix
Table E.3: Additional backbone experiments with Bi-SFT on 100k samples followed by bidirectional RL on 10k samples. Scores are accuracy (%); Avg. is the mean over the four displayed benchmarks. Bold indicates the higher value within each backbone pair.
Method
Generation basis
On-policy generation
Cycle verification target
Bidirectional RL
Training
WinClick ( Hui et al. 2025 )
Annotations
×
—
×
SFT
Look Where You’re Told ( Kim et al. 2026 )
Attention
×
Attention distribution
×
SFT
UI-Ins ( Chen et al. 2025 )
—
×
—
×
SFT+GRPO
BiGrounder (Ours)
Coordinates
✓
GT location
✓
SFT+GRPO
Appendix
Table F.1: Qualitative comparison of instruction generation and grounding. On-policy generation and bidirectional RL refer to an independent instruction-generation task, excluding UI-Ins’s instruction-as-reasoning. Annotations denotes labelled element information without specifying its input encoding. GT: ground truth.
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · VCIP & TMCC & DISSec, College of Computer Science, Nankai University