RL for GUI Agents

RL: Reinforcement Learning · GUI: Graphical User Interface

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 21

All topics

When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction
Oct 8, 2026Xiaolong Li, Xiaohan Xu, Jinyang Li +5Human-AI Interaction LLM Agent Harnesses
The University of Hong Kong · The BIRD Team · Microsoft Research
SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents
Sep 29, 2026Shengtian Yang, Ziyu Xiong, Kaibing Yang +6Reinforcement Learning RL for GUI Agents
Southeast University, Nanjing, China · Kuaishou Technology, Beijing, China · Unaffiliated
CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications
Aug 12, 2026Linqiang Guo, Li Gu, Zihuan Jiang +8Reinforcement Learning RL for GUI Agents
Peter · Concordia University · Mila – Québec AI Institute +2
Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification
Jul 28, 2026Chenrui Shi, Yuwei Wu, Yang Liu +5RL for GUI Agents Automated Evaluation
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, Shenzhen, China +1
SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents
Jul 25, 2026Yang Wan, Zhenhao Zhang, Jierui Wang +1Reward Modeling LLM-as-a-Judge
1Zhejiang University · University of Rochester
EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Jul 7, 2026Mianqiu Huang, Taofeng Xue, Chong Peng +12RL for GUI Agents Partially Observable RL
1Meituan · 2Fudan University · 3Shanghai Jiao Tong University +1
What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States
Jun 30, 2026Chen Liu, Ling Chen, Hanzhang Zhou +7RL for GUI Agents GUI Agents
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
Xiaomi-GUI-0 Technical Report
Jun 30, 2026Wanxia Cao, Chengzhen Duan, Pei Fu +29Mobile GUI Automation RL for GUI Agents
GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
Jun 29, 2026Sunqi Fan, Lingshan Chen, Runqi Yin +4RL for GUI Agents GUI Agents
Tsinghua University · Tencent Hunyuan
Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation
Jun 23, 2026Marta Sumyk, Oleksandr KosovanRL for GUI Agents RL Fine-Tuning
Ukrainian Catholic University, Lviv, Ukraine
ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents
Jun 22, 2026Yincheng Zhou, Athena Zhuoming Zhong, Shijie Zhang +3Distribution Shift Robustness Computer-Use Agent Benchmarks
University of Pennsylvania · 2Peking University
VISTA: View-Consistent Self-Verified Training for GUI Grounding
Jun 12, 2026Xinyu Qiu, Yunzhu Zhang, Heng Jia +3Multi-View Consistency VLM Robustness
Zhejiang University · Venus Team, Ant Group
GUI-AC: Enhancing Continual Learning in GUI Agents
Jun 9, 2026Can Lin, Tao Feng, Hangjie Yuan +3Non-Stationary RL RL for GUI Agents
Beijing University of Posts and Telecommunications · Tsinghua University · Zhejiang University +1
StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents
Jun 5, 2026Haojie Hao, Longkun Hao, Yihang Lou +8Reinforcement Learning RL for GUI Agents
Beihang University · Peking University · Renmin University of China +4
GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
May 29, 2026Junlong Li, Chao Hao, Lap-Pui Chau +1Multimodal Grounding Agentic RL
The Hong Kong Polytechnic University
Darwin Mobile Agent: A Roadmap for Self-Evolution
May 26, 2026Daniel Beechey, Derek Yuen, Jianheng Liu +5Reinforcement Learning RL for GUI Agents
1Huawei Noah’s Ark Lab · University College London
ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
May 12, 2026Xuhao Hu, Xi Zhang, Haiyang Xu +6Agentic RL RL for GUI Agents
Tongyi Lab, Alibaba Group · Fudan University · Shanghai Artificial Intelligence Laboratory
LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning
May 8, 2026Yubin Wu, Zicheng Cai, Liping Ning +4RL for GUI Agents GUI Agents
Moore Threads AI
Faithful Mobile GUI Agents with Guided Advantage Estimator
May 2, 2026Haowen Hu, Pengzhou Cheng, Zheng Wu +3Reinforcement Learning RL for GUI Agents
1Shanghai Jiao Tong University.
GUI Agents with Reinforcement Learning: Toward Digital Inhabitants
Apr 30, 2026Junan Hu, Jian Liu, Jingxiang Lai +6Reinforcement Learning RL for GUI Agents
1Shandong University · 2The Hong Kong University of Science and Technology · 3The Hong Kong University +2
SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning
Apr 24, 2026Jichao Wang, Liuyang Bian, Yufeng Zhou +9RL for GUI Agents Credit Assignment in RL
1vivo AI Lab · 2Zhejiang Lab · 5Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences +2

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction

Xiaolong Li, Xiaohan Xu, Jinyang Li, Xinnuo Xu+4

The University of Hong Kong · The BIRD Team · Microsoft Research

Human-AI Interaction LLM Agent Harnesses RL for GUI Agents AI Agent Benchmarks

Open paper AskPDF arXiv

Most human-agent interaction today remains text-based. Natural language can impose cognitive overload, ambiguity, information chaos, and slow input for complex tasks; ephemeral generative UIs can present structured information and guide users toward task completion. We propose GenUI-Harness, a multi-agent harness pairing a Tool Agent for information retrieval and task execution with a GUI Coder Agent that identifies ambiguities and generates front-end code for structured interfaces. Training the coder with reinforcement learning is challenging: verifiable rewards for interactive UI generation require costly execution, while LLM-as-a-Judge rewards are prone to reward hacking. We address the first challenge with Dynamic UX, a lightweight package for dynamic interaction and reward collection in a single sandbox, and the second with Reward Auditor, a meta-reward mechanism that monitors reward distributions and distills diagnostic patterns into a shared rubric and scoring specification. We introduce UI-TAU Bench, a benchmark for active human-agent interaction through generated UI code, built on 10 real-world domain databases constructed from public data sources and based on Tau-Bench tool-use settings, with Lite (300 tasks) and Full (1,000 tasks) splits. GenUI-Harness achieves an average Pass@3 gain of 4.48 percentage points over smolagents on Lite. Training with GenUI-Harness improves a 4B backbone from 9.33% to 58.00% Pass@3, outperforming larger frontier models such as Claude Opus 5 (46.67%). GenUI-Harness also remains robust on ambiguous and non-ambiguous queries. In a reviewer survey comparing communication channels, generated UIs reduce average dialogue rounds from 3.4 to 1.2. These results show that data-aware generative interfaces can support effective task completion and reduce dialogue rounds in evaluated database-backed workflows.

Momentum

Related topics

Latest papers 21