BiGrounder: Bidirectional Training Unlocks the Potential of GUI Grounding
Authors: Dengyao Shi, Tianyu Fu, Fei Hu, Anyang Su, Hanning Wang, Shuo Wang, Minghui Wu, Chenxu Zhao, Zhen Lei
Organizations: MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Mininglamp Technology
Graphical User Interface (GUI) grounding aims to map natural language instructions to precise operational coordinates, serving as a critical task for autonomous agent interaction. Current approaches formalize GUI grounding as predicting action-coordinate pairs from instructions, with end-to-end training supervised solely by final coordinates. Training datasets typically provide only a small number of descriptions per target coordinate, whereas real-world scenarios involve a much larger set of varied descriptions for the same target, which limits robustness to linguistic variation. While recent GUI grounding paradigms have begun employing reinforcement learning to enhance localization accuracy, models with limited exploratory capability before reinforcement yield constrained improvements even with identical training data during the reinforcement phase. To overcome these limitations, we propose a bidirectional reward paradigm that enables model self-reinforcement. During SFT, bidirectional training equips the model to generate action descriptions from target coordinates. During RL, the model samples semantically consistent instruction variations on-policy and grounds each variation back to its target. A cycle-consistency gate couples semantic quality to grounding success, turning the model's own instruction policy into a source of continually refreshed reinforcement trajectories. Our 4B-parameter model achieves competitive accuracy across five grounding benchmarks in our evaluation, indicating the effectiveness of the proposed training paradigm. https://github.com/Spell-shi/BiGrounder
Figures & tables
Figure 1: Existing datasets (a) suffer from limited instruction coverage, whereas our bidirectional reinforcement framework (b) samples on-policy instruction variations for the same UI element, strengthening semantic–element understanding.
Figure 2: During the SFT stage, we organize the training data into six tasks: instruction generation, instruction grounding, bounding-box description generation, bounding-box localization, point-based description generation, and point localization.
Figure 3: The reinforcement stage couples on-policy instruction-generation trajectories with their corresponding instruction-grounding trajectories.
Model
ScreenSpot-Pro
ScreenSpot-V2
MMBench-GUI L2
UI-Vision
OSWorld-G-R
Avg.
Qwen2.5-VL-3B backbone
Qwen2.5-VL-3B ∗ ( Bai et al. 2025b )
25.9
80.8
52.0
0.7
39.4
39.8
UI-R1-E-3B ( Lu et al. 2025 )
17.8
88.6
68.4
16.5
48.8
48.0
SE-GUI-3B ( Yuan et al. 2025 )
35.9
86.8
66.3
15.0
46.1
50.0
InfiGUI-R1-3B ( Liu et al. 2025b )
35.7
89.5
70.6
17.8
42.9
51.3
GUI-G 2 -3B ( Tang et al. 2025 )
36.4
87.6
66.3
18.7
53.5
52.5
Table 1: Grounding accuracy on five GUI grounding benchmarks and their average, grouped by backbone. Unless marked with “ ∗ ”, baseline results are taken from the corresponding original papers or technical reports (cited in each row), or from the consolidated evaluation reported by GroundNext ( Feizi et al. 2025 ) . “ ∗ ” marks results we re-evaluated under our unified protocol for benchmarks not reported in these sources. “Avg.” is the mean over the five benchmarks. Within each backbone group, best is in bold and second-best is underlined. The green subscript denotes the absolute Avg. gain over each model’s backbone.
Executor Model
AndroidWorld
OSWorld
Aria-UI ( Yang et al. 2024 )
44.8
15.2
UGround-V1-7B ( Gou et al. 2025 )
32.8
16.1
UI-TARS-1.5-7B ( Seed 2025 )
37.9
19.1
ScaleCUA-7B ( Liu et al. 2025c )
48.3
28.1
Qwen3-VL-4B ∗ ( Bai et al. 2025a )
68.9 ± 0.9
40.2 ± 0.8
BiGrounder-4B
72.4 ± 0.9
42.8 ± 0.5
Table 2: GUI agent benchmark results with each model as the grounding executor and GPT-4o as the planner. AndroidWorld follows predefined steps; OSWorld uses a 50-step budget. Best in bold. ∗ Evaluated by us (mean ± std over 3 runs).
Method
Original Accuracy ↑
Any Correct ↑
All Correct ↑
Any–All Gap ↓
Qwen3-VL-4B
65.3
73.4
56.0
17.4
Bi-SFT
65.7
73.6
56.8
16.8
Bi-SFT + Forward RL
67.4
74.5
58.2
16.3
BiGrounder-4B
69.5
75.3
60.5
14.8 ↓2.6
Table 3: Robustness to instruction variation on OSWorld-G-R-Variants . Each original instruction is paired with three target-preserving variants that differ in wording and sentence structure. Original : accuracy under the original instruction; Any-/All-Correct : element grounded correctly under at least one/all four instructions. Any–All Gap : sensitivity to instruction variation. The green subscript denotes the absolute gap reduction over Qwen3-VL-4B. Best in bold.
Figure 4: RL training dynamics. (a) semantic consistency (BERTScore) and within-group diversity ( 1− Self-BLEU) of the on-policy instructions. (b) cycle-grounding reliability measured by Any@4 and All@4, with the shaded Any–All gap narrowing over training.
Figure 5: Qualitative comparison of grounding under instruction variations.
Method
SS-Pro
SS-V2
OS-G-R
UI-V.
MM-L2
Avg.
Qwen3-VL-4B
59.3
93.5
65.3
26.9
81.2
65.2
Single-SFT
59.4
90.6
65.2
39.7
79.8
66.9 +1.7
Bi-SFT
60.2
91.2
65.7
39.9
81.0
67.6 +2.4
+RL-InBbox
63.2
92.9
67.4
44.9
84.1
70.5 +5.3
+RL-Distance
62.7
92.3
66.8
42.5
83.5
69.6 +4.4
+RL-Gaussian
63.5
92.9
67.9
44.8
85.1
70.8 +5.6
Table 4: Controlled comparison of training recipes with Qwen3-VL-4B-Instruct backbone and complete training dataset. The subscript in Avg. denotes the absolute gain over the base model.
Method
SS-Pro
SS-V2
OS-G-R
UI-V.
MM-L2
Avg.
Bi-SFT
60.2
91.2
65.7
39.9
81.0
67.6
RL w/o Cycle-C.
62.8
92.4
66.9
44.9
83.5
70.1 +2.5
RL w/ Cycle-C.
66.6
93.9
69.5
46.6
87.5
72.8 +5.2
Table 5: Ablation on cycle-consistency gating for mitigating reward hacking. The subscript in Avg. denotes the absolute gain over the Bi-SFT baseline.
Method
SS-Pro
SS-V2
OS-G-R
UI-V.
MM-L2
Avg.
Qwen3-VL-4B
59.3
93.5
65.3
26.9
81.2
65.2
Bi-SFT w/o rewritten
59.7
91.2
65.5
39.5
80.6
67.3
Bi-SFT w/ rewritten
60.2
91.2
65.7
39.9
81.0
67.6
BiGrounder w/o rewritten
64.7
93.1
68.1
44.8
85.6
71.3
BiGrounder w/ rewritten
66.6
93.9
69.5
46.6
87.5
72.8
Table 6: Effect of Gemini-rewritten instructions under the same backbone and training recipe.
Figure 6: Training dynamics under different reward hyperparameters: (a) semantic-reward threshold t ; (b) pixel-level tolerance τ .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Instruction following comparison before and after cycle-consistency reinforcement, showing improved spatial grounding.
Figure A.2: Instruction following comparison before and after cycle-consistency reinforcement, showing improved icon grounding.
Figure A.3: Analyzing the semantic generation capability before and after reinforcement learning.
Figure B.1: Composition of the 4,482k-pair training corpus. The inner ring splits the data into forward (instruction → coordinate) and reverse (coordinate → instruction) halves of equal size; the outer ring breaks each half down by data source and grounding modality. The GroundCUA portion uses Video-Audio Production samples and their reverse counterparts. As an optional augmentation, the 790k Grounding bbox samples—those that already carry bounding-box annotations—have their instructions rewritten with Gemini-2.5-Flash to broaden localization-based reward coverage during RL; the core pipeline can be trained on open-source data alone.
Filter
Kept (C / I)
Precision ↑
NFR ↑
No filter
273 / 27
91.0% (273/300)
0.0% (0/27)
BERTScore >t
268 / 22
92.4% (268/290)
18.5% (5/27)
LLM Judge
267 / 7
97.4% (267/274)
74.1% (20/27)
Cycle-Consistency
260 / 2
99.2% (260/262)
92.6% (25/27)
Appendix
Table D.1: Noise filtering on 300 manually annotated reverse instructions, including 273 correct and 27 incorrect ones. Precision : fraction of correct samples in the kept set. Noise Filtering Rate (NFR) : percentage of incorrect samples successfully filtered out.
Reverse Instruction
BERTScore
Cycle-C.
Reward
Click the star to follow the repository.
0.86
✗
0.00
Click the star to bookmark this webpage.
0.92
✓
0.92
Appendix
Table D.2: Case study showing that Cycle-Consistency filters out high-BERTScore but visually incorrect reverse instructions. GT: “Click star to save this page.”
Method
Rev.
Fwd.
SS-Pro
SS-V2
OS-G-R
MM-L2
Avg.
Forward-only RL ( 1×16 )
0
16
63.7
92.9
67.4
85.3
77.3
Forward-only RL ( 1×20 )
0
20
63.9
92.9
67.9
85.1
77.5
BiGrounder Gate-only
4
16
65.6
92.9
68.6
86.2
78.3
BiGrounder
4
16
66.6
93.9
69.5
87.5
79.4
Appendix
Table E.1: Rollout-budget and reverse-reward controls. Rev./Fwd.: reverse/forward trajectories per target. SS-Pro: ScreenSpot-Pro; SS-V2: ScreenSpot-V2; OS-G-R: OSWorld-G-Refine; MM-L2: MMBench-GUI L2. Scores are accuracy (%). Avg. is the unweighted mean over these four benchmarks, excluding UI-Vision. The 1×20 baseline matches BiGrounder’s total trajectory count. Gate-only uses binary reverse rewards.
Method
Original
Any ↑
All ↑
Gap ↓
Forward-only RL
67.4
74.5
58.2
16.3
Offline aug.
68.2
75.0
59.1
15.9
Online aug.
68.8
75.2
60.1
15.1
BiGrounder
69.5
75.3
60.5
14.8
Appendix
Table E.2: Instruction-generation and reverse-update ablations on OSWorld-G-Refine-Variants. Original, Any, and All are accuracies (%); Gap is Any minus All in percentage points. Offline/online augmentation use a frozen Bi-SFT/current-policy instruction generator, respectively, with forward-only updates; BiGrounder also updates the reverse policy. These three methods share rollout budgets and optimization-step counts.
Method
SS-Pro
SS-V2
OS-G-R
MM-L2
Avg.
Qwen3.5-9B
65.2
91.8
67.7
80.4
76.3
+ BiGrounder
65.7
92.0
68.0
85.1
77.7
Qwen3.5-35B-A3B
64.5
95.2
63.6
87.5
77.7
+ BiGrounder
65.3
94.7
67.6
88.1
78.9
Appendix
Table E.3: Additional backbone experiments with Bi-SFT on 100k samples followed by bidirectional RL on 10k samples. Scores are accuracy (%); Avg. is the mean over the four displayed benchmarks. Bold indicates the higher value within each backbone pair.
Method
Generation basis
On-policy generation
Cycle verification target
Bidirectional RL
Training
WinClick ( Hui et al. 2025 )
Annotations
×
—
×
SFT
Look Where You’re Told ( Kim et al. 2026 )
Attention
×
Attention distribution
×
SFT
UI-Ins ( Chen et al. 2025 )
—
×
—
×
SFT+GRPO
BiGrounder (Ours)
Coordinates
✓
GT location
✓
SFT+GRPO
Appendix
Table F.1: Qualitative comparison of instruction generation and grounding. On-policy generation and bidirectional RL refer to an independent instruction-generation task, excluding UI-Ins’s instruction-as-reasoning. Annotations denotes labelled element information without specifying its input encoding. GT: ground truth.
Existing agentic reinforcement learning methods for GUI grounding have limitations at two levels. At the data level, current approaches typically treat all training samples equally, although their training value to the baseline model varies with difficulty. Overlooking this can greatly reduce training efficiency or even cause collapse. At the strategy level, existing frameworks struggle to balance the trade-off between cropping larger regions for sufficient context and smaller ones for reduced redundancy, a tension inherent to tool-augmented grounding agents. In addition, overly complex decision-making is difficult for small-parameter models and significantly increases inference time. To address these issues, at the data level, we propose GUI-D, a data mining and difficulty scoring pipeline that identifies the training-worthy samples by proper testing and assigns difficulty scores to guide subsequent training weights. At the strategy level, we propose GUI-C2, which employs an area-gated coarse-to-fine refinement mechanism that progressively narrows the visual field via model-internal uncertainty signals, adaptively reserving context for large targets while amplifying precision for small ones, reinforced by improvement-aware stage rewards that ensure each refinement genuinely advances grounding. Meanwhile, we simplify the decision-making process to greatly reduce additional inference time. Finally, extensive experiments show that our method achieves state-of-the-art performance. The code and data will be publicly available.
Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.
Yan Zhang, Daiqing Wu, Huawen Shen +2
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · VCIP & TMCC & DISSec, College of Computer Science, Nankai University
Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen. Existing Multimodal Large Language Model (MLLM) approaches typically formulate GUI grounding as a text-based coordinate generation task. However, directly generating precise coordinates from visual inputs is challenging and often data-intensive. A more intuitive strategy is to first identify instruction-relevant visual patches and then determine the exact click location within them. Motivated by recent observations that general MLLMs exhibit native grounding ability embedded in their attention maps, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 509k samples (around 101k screenshots), demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 61.5% on ScreenSpot-Pro, 92.1% on ScreenSpot-v2, 68.1% on OSWorld-G, 79.1% on MMBench-GUI-L2, and 60.0% on UI-Vision. Project page: https://github.com/sjz5202/GUI-AIMA .