LLM Agent Verification

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 260% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 72

All topics

OpenComputer: Verifiable Software Worlds for Computer-Use Agents
May 19, 2026Jinbiao Wei, Qianran Ma, Yilun Zhao +4Computer-Use Agent Benchmarks Computer-Use Agents
Yale NLP Lab · University of Pennsylvania · University of North Carolina at Chapel Hill
Verify-Gated Completion as Admission Control in a Governed Multi-Agent Runtime: A Bounded Architecture Case Study
May 18, 2026Hai-Duong Nguyen, Xuan-The TranLLM Agent Verification Multi-Agent Systems
Vietnam Maritime University, Haiphong, Vietnam
Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps
May 17, 2026Tanmay Asthana, Aman Saksena, Divyansh SahuAI Agent Reliability Deep Research Agents
Deccan AI Research
No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills
May 13, 2026Ying Li, Hongbo Wen, Yanju Chen +3LLM Guardrails LLM Agent Verification
University of California, Los Angeles · University of California, Santa Barbara · University of California, San Diego
Behavioral Integrity Verification for AI Agent Skills
May 12, 2026Yuhao Wu, Tung-Ling Li, Hongliang LiuAgent Reliability AI Agent Auditing
Palo Alto Networks
Methods for Formal Verification of Agent Skills: Three Layers Toward a Mechanically Checkable Capability-Containment Proof
May 9, 2026Alfredo MetereLLM Agent Security Formal Verification
Metere Consulting, LLC
MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents
May 7, 2026Ashwani Anand, Ivi Chatzi, Ritam Raha +1Computer-Use Agent Benchmarks AI Agent Reliability
Max Planck Institute for Software Systems, Kaiserslautern, Germany
Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes
May 1, 2026Alfredo MetereAI Agent Security LLM Agent Verification
Enclawed, LLC, California, USA
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
Apr 1, 2026Alibek Kaliyev, Artem MaryanskyyAI Agent Reliability Tool-Use Evaluation
The University of Texas at Austin · Austin, Texas, USA · Uber +1
When Is Enough Not Enough? Illusory Completion in Search Agents
Feb 7, 2026Dayoon Ko, Jihyuk Kim, Sohyeon Kim +5AI Agent Reliability LLM Answer Verification
Seoul National University · LG AI Research · University of Illinois Chicago
Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry
Oct 29, 2025Run Peng, Ziqiao Ma, Amy Pang +5Multi-Agent LLM Systems LLM Agent Verification
University of Michigan · UNC, Chapel Hill · Georgia Tech +2

May 19, 2026, cs.AIJ/K move · Enter open · S save

OpenComputer: Verifiable Software Worlds for Computer-Use Agents

Jinbiao Wei, Qianran Ma, Yilun Zhao, Xiao Zhou+3

Yale NLP Lab · University of Pennsylvania · University of North Carolina at Chapel Hill

Computer-Use Agent Benchmarks Computer-Use Agents AI Agent Evaluation LLM Agent Verification Automated Evaluation

Open paper AskPDF arXiv

We present OpenComputer, a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer's hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation.

Momentum

Related topics

Latest papers 72