cs.CVMar 12, 2026
SaveHATS: Hardness-Aware Trajectory Synthesis for GUI Agents
Organizations: Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute · Huawei Noah’s Ark Lab
Abstract
Graphical user interface (GUI) agents powered by large vision-language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantically ambiguous actions, whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) hardness-driven exploration, which guides data collection toward ambiguous yet informative interactions, and (2) alignment-guided refinement, which iteratively validates and repairs instruction-execution alignment. The two modules operate in a closed loop: exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments.
Figures & tables
Figure 1: Overview of trajectory synthesis paradigms. Compared with (a) existing methods, (b) HATS integrates hardness-driven exploration and alignment-guided refinement in a closed loop, producing high-quality trajectories with rich semantic coverage and strong instruction–execution alignment. (c) Experiments show HATS outperforms OS-Genesis by 100%↑ on AndroidWorld ( 22.60 vs. 11.30 ) and 215%↑ on WebArena ( 20.60 vs. 6.53 ).
Figure 2: Illustrative cases of semantic-ambiguous actions. Such actions constitute critical bottlenecks for robust agent generalization but are rarely captured by existing synthesis pipelines.
Figure 3: Architecture of the HATS framework. The framework integrates a Hardness-Driven Exploration module (§ 3.1 ) and an Alignment-Guided Refinement module (§ 3.2 ) within a unified HD-MCTS loop. Exploration corresponds to the Selection , Expansion , and Simulation Phase I , while refinement handles Simulation Phase II and Backpropagation . Misalignment detected during refinement is converted into a hardness reward that guides subsequent exploration, forming a closed loop for progressively improving both diversity and semantic fidelity of synthesized trajectories.
Figure 4: Comparison of exploration strategies. Uniform Random Exploration [ 59 ] often yields trivial and redundant actions, whereas our Hardness-Driven Exploration replaces random walking with a hardness-driven exploration policy that selectively targets under-represented yet semantically challenging actions.
Figure 5: Comparison of instruction synthesis methods. One-Shot Instruction Generation [ 59 ] directly maps raw traces to text, often yielding vague or underspecified goals and inconsistent executions. In contrast, our Multi-Round Alignment-Guided Refinement iteratively replays and verifies task instructions to produce semantically faithful and executable Verified Trajectories.
| Base Model | Strategies | Task Categories (%) | Overall SR (%) | |||
|---|---|---|---|---|---|---|
| DLS | S&C | S&U | P&W | |||
| GPT-4o [ 20 ] | Zero-Shot [ 63 ] | 42.86 N.A. | 66.67 N.A. | 77.27 N.A. | 29.09 N.A. | 45.22 N.A. |
| InternVL2-4B [ 4 ] | Zero-Shot [ 63 ] | 0.00 -100.0% | 0.00 -100.0% | 2.38 -94.2% | 0.00 -100.0% | 1.02 -95.8% |
| Task-Driven [ 25 ] | 3.49 -79.8% | 0.00 -100.0% | 19.70 -51.8% | 3.70 -57.2% | 7.90 -65.1% | |
| Task-Driven w. Self-Instruct [ 62 ] | 2.30 -86.7% | 3.70 -90.4% | 21.21 -48.2% | 0.62 -92.8% | 6.09 -73.4% | |
| OS-Genesis [ 59 ] | 7.14 -58.6% | 11.11 -71.2% | 36.36 -11.1% | 3.64 -57.9% | 11.30 -50.0% | |
Table 1: AndroidWorld evaluation results across task categories. All values are success rates (%). DLS = Daily Life & Services; S&C = Social & Communication; S&U = System & Utility; P&W = Productivity & Work. The final column shows overall success rate.
| Base Model | Strategies | WebArena (Success Rate %) | |||
|---|---|---|---|---|---|
| Gitlab | Maps | Overall | |||
| GPT-4o [ 20 ] | Zero-Shot [ 63 ] | 32.14 N.A. | 6.42 N.A. | 27.36 N.A. | 22.82 N.A. |
| InternVL2-4B [ 4 ] | Zero-Shot [ 63 ] | 1.11 -95.1% | 0.00 -100.0% | 0.94 -94.8% | 0.76 -96.3% |
| Task-Driven [ 25 ] | 9.52 -58.1% | 5.00 -75.2% | 0.00 -100.0% | 5.72 -72.2% | |
| Task-Driven w. Self-Instruct [ 62 ] | 9.52 -58.1% | 7.50 -67.0% | 0.00 -100.0% | 6.41 -68.9% | |
| OS-Genesis [ 59 ] | 7.94 -65.1% | 7.50 -67.0% | 3.13 -82.6% | 6.53 -68.3% | |
Table 2: WebArena evaluation results across website domains. All values are success rates (%).
| Hardness Metric | Hyperparameters | Success Rate (%) | |
|---|---|---|---|
| 0.01 | 0.50 | 22.7 -55.3% | |
| 0.01 | 1.00 | 50.8 0.0% | |
| 0.01 | 2.00 | 39.2 -22.8% | |
| 0.10 | 0.50 | 13.8 -72.8% | |
| 0.10 | 1.00 | 40.0 -21.3% | |
Table 3: Parameter sensitivity of the hardness reward in Eq. 3 .
| Alignment Metric Definition in Eq. 2 | Success Rate (%) |
|---|---|
| 50.8 | |
| 36.0 |
Table 4: Metric comparison of action-level alignment.
Figure 6: Performance comparison & corpus composition. Top: performance comparison on AndroidWorld using InternVL2-4B. Bottom: corpus composition by task category for the baseline (left) and HATS (right), where HATS increases exploration in categories that are empirically harder for existing methods (e.g., P&W ), while reducing exploration in easier categories (e.g., S&U ).
Figure 7: Radar chart comparison. (a) Action-type distribution on WebArena , showing distributional differences between the baseline and HATS . (b) Per-task view of multi-round refinement, where the shaded area expands across rounds, indicating higher action-level recall and better synthesized trajectories.
Figure 8: Semantic-ambiguity ratios at log scale. HATS vs. OS-Genesis across ambiguity types Vis, Seq, and Ctx on AndroidWorld (AW) and WebArena (WA). Bars correspond to AW/WA-Ctx/Seq/Vis pairs. Trajectories may involve multiple ambiguity types, so categories are not mutually exclusive.
| Refinement Rounds | Success Rate (%) | |
|---|---|---|
| One-Shot | 0.26 -35.0% | 16.00 -30.6% |
| Round 1 | 0.19 -52.5% | 15.56 -32.6% |
| Round 2 | 0.29 -27.5% | 21.43 -7.1% |
| Round 3 | 0.40 0.0% | 23.07 0.0% |
Table 5: Ablation on multi-round refinement on AndroidWorld.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Application | Category | Application | Category |
|---|---|---|---|
| Calculator | P&W | VLC | DLS |
| Calendar | P&W | Markor | P&W |
| Clock | S&U | RetroMusic | DLS |
| Contacts | S&U | Gallery | DLS |
| Dialer | S&U | Tasks | P&W |
| Settings | S&U | SMSMessenger | S&C |
Table 6: Application categorization for AndroidWorld.
| Model | Learning Rate | Batch Size | Epochs |
|---|---|---|---|
| InternVL2-4B | 4e-5 | 128 | 15 |
| InternVL2-8B | 4e-5 | 128 | 15 |
| Qwen2-VL-7B | 2e-6 | 128 | 15 |
Table 7: Training hyperparameters for VLM backbones.
| Task ID | Round | Synthesized Instruction |
|---|---|---|
| Task 1 | 1 | In the Audio Recorder app, rename the audio recording titled ’05-06-2025 06.14.28PM’ to ’Meeting_Notes_05-06-2025’. First, double-click the text field containing ’05-06-2025 06.14.28PM’ to select it. Then, double-click the ’Copy’ option from the edit menu. Double-click the date and time field again to ensure focused editing, and double-click the ’Paste’ option from the edit menu. Finally, input ’Meeting_Notes_05-06-2025’ into the text field to set the new name. |
| 2 | In the Audio Recorder app, rename the audio recording titled ’05-06-2025 06.14.28PM’ to ’Meeting_Notes_05-06-2025’. First, locate the text field containing ’05-06-2025 06.14.28PM’ in the renaming prompt and double-click it to select the text. Then, double-click the ’Copy’ option from the edit menu above the selected text. Next, double-click the same text field again to ensure it is focused for editing. After that, click the ’Paste’ option from the edit menu to paste the copied text. Finally, input ’Meeting_Notes_05-06-2025’ into the text field to set the new name, and click the ’Save’ button to confirm the changes. | |
| 3 | In the Audio Recorder app (package name: com.dimowner.audiorecorder), properly set up the renaming interface to change the audio recording titled ’05-06-2025 06.14.28PM’ to ’Meeting_Notes_05-06-2025’. Start by double-clicking the text field containing ’05-06-2025 06.14.28PM’ to select it in the renaming prompt. Then, double-click the ’Copy’ option from the edit menu above the selected text to copy it. Double-click the text field to refocus it if necessary. Click the ’Paste’ option from the edit menu to paste the copied text. Finally, manually input ’Meeting_Notes_05-06-2025’ into the text field for renaming and press the ’Save’ button to confirm the change. Ensure that you are on the renaming dialog screen directly accessing the desired recording and follow the sequence precisely. | |
| Task 2 | 1 | Delete the audio records ’Record-10’ and ’Record-4’ in the Audio Recorder app. |
| 2 | In the Audio Recorder app, first tap on the bookmark icon next to ’Record-10’ to select it, then double-tap on ’Record-4’ to select it. Finally, tap on the trash bin icon in the top-right corner to delete the selected records. | |
| 3 | In the Audio Recorder app, first tap on the bookmark icon next to ’Record-10’ to select it. Then scroll down if necessary to ensure ’Record-4’ is visible. Once visible, double-tap on ’Record-4’ to select it by quickly tapping twice on the bookmark icon next to ’Record-4’. After both ’Record-10’ and ’Record-4’ are selected, tap on the trash bin icon in the top-right corner to delete these selected records. |
Table 8: Evolution of synthesized instructions across three refinement rounds for the tasks in Figure 7 b.
More in this topic
LLM Agent Trajectory Synthesis
LLM Agent Trajectory Synthesis creates and validates interaction sequences to train and improve language-model agents.
See all 68 papers