LLM Agent Harnesses

LLM: Large Language Model

An LLM agent harness surrounds a language model with the mechanisms needed to act: tool interfaces, control loops, context and memory management, persistent state, and execution environments. Common approaches include ReAct-style loops, explicit workflow graphs, typed tool contracts, sandboxes, and verification or recovery steps; newer systems also adapt harnesses using execution traces, search, or reinforcement learning. Researchers evaluate these systems on coding, web, terminal, scientific, and interactive tasks, measuring not only success but also cost, reliability, and safety. A central tension is how much structure helps: added guidance can improve control and recovery, yet may constrain useful behavior, fail to transfer across models or tasks, or make comparisons hard to interpret without specifying the harness.

Latest papers

389 papers118 papers in the last four weeks, up 103% on the four weeks before. 0.7% of all new papers.

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
May 28, 2026Minhua Lin, Juncheng Wu, Zijun Wang +14LLM Agent Evaluation Agent Harness Optimization
1The Pennsylvania State University · 2UC Santa Cruz · 3Amazon +3
Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +10Computer-Use Agent Benchmarks AI Agent Reliability
Peking University · Qiyuan Tech
FinHarness: An Inline Lifecycle Safety Harness for Finance LLM Agents
May 26, 2026Haoxuan Jia, Yang Liu, Bin Chong +10AI Agent Reliability LLM Guardrails
Nanyang Technological University · Tsinghua University · Peking University +8
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
May 26, 2026Yong-eun ChoAI Agent Reliability LLM Agent Evaluation
KailosLab · Seoul, Republic of Korea
Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents
May 21, 2026Tianshi Xu, Huifeng Wen, Meng LiLLM Agents LLM Agent Harnesses
Peking University
Code as Agent Harness
May 18, 2026Xuying Ning, Katherine Tieu, Dongqi Fu +39LLM Agent Orchestration Long-Horizon LLM Agents
Harnessing LLM Agents with Skill Programs
May 18, 2026Hongjun Liu, Yifei Ming, Shafiq Joty +1LLM Agent Self-Improvement LLM Agent Skill Learning
New York University · Salesforce AI Research
Harnesses for Inference-Time Alignment over Execution Trajectories
May 15, 2026Boyuan Wang, Bochao Li, Minghan Wang +2LLM Planning Agent Harness Optimization
Southern University of Science and Technology
Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
May 14, 2026Sahil Sen, Akhil Kasturi, Elias Lumer +2Agentic Search LLM Agent Evaluation
PricewaterhouseCoopers, U.S.
Harness Engineering as Categorical Architecture
May 12, 2026Bogdan BanuLLM Agent Orchestration LLM Agent Harnesses
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses AI Agent Benchmarks
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
Continual Harness: Online Adaptation for Self-Improving Foundation Agents
May 11, 2026Seth Karten, Joel Zhang, Tersoo Upaa +5Continual Learning for LLM Agents Game-Playing Agents
Princeton University · ARISE Foundation · Google DeepMind
Workspace Optimization: How to Train Your Agent
May 10, 2026Elad Sarafian, Gal Kaplun, Ron Banner +2Multi-Agent LLM Systems LLM Agent Self-Improvement
NVIDIA · NVIDIA & Technion
Stop Comparing LLM Agents Without Disclosing the Harness
May 7, 2026Yunbei Zhang, Janet Wang, Yingqiang Ge +3LLM Agent Evaluation Long-Horizon Agent Evaluation
Tulane University · Rutgers University · Independent Researcher +1
AgenticRAG: Agentic Retrieval for Enterprise Knowledge Bases
May 7, 2026Susheel Suresh, Hazel Mak, Shangpo Chou +2Tool-Augmented Language Model Agents Agentic Search
Microsoft Corporation
Design Conductor 2.0: An agent builds a TurboQuant inference accelerator in 80 hours
May 6, 2026The Verkor Team, Ravi Krishna, Suresh Krishna +1AI Accelerator Inference Multi-Agent LLM Systems
The Verkor Team
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Apr 28, 2026Jiahang Lin, Shichun Liu, Chengjun Pan +8AI Coding Agents Agent Harness Optimization
Fudan University · Shanghai Qiji Zhifeng Co., Ltd · Peking University
The Last Harness You'll Ever Build
Apr 22, 2026Haebin Seong, Li Yin, Haoran Zhang +1Meta-Learning Agent Harness Optimization
HadAgent: Harness-Aware Decentralized Agentic AI Serving with Proof-of-Inference Blockchain Consensus
Apr 15, 2026Landy Jimenez, Mariah Weatherspoon, Bingyu Shen +3LLM Agent Harnesses AI Agent Security
Department of Computer Science and Technology, Kean University, Union, New Jersey, U.S. · Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, Indiana, U.S. · Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, Florida, U.S. +1
SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment
Apr 15, 2026Xixun Lin, Yang Liu, Yancheng Chen +8LLM Agent Safety LLM Agent Harnesses
Institute of Information Engineering, Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · Institute of Applied Physics and Computational Mathematics +1
AgentSPEX: An Agent SPecification and EXecution Language
Apr 14, 2026Pengcheng Wang, Jerry Huang, Jiarui Yao +7LLM Agent Orchestration LLM Agent Harnesses
University of Illinois Urbana-Champaign · Baylor College of Medicine · University of Wisconsin–Madison
SemaClaw: A Step Towards General-Purpose Personal AI Agents through Harness Engineering
Apr 13, 2026Ningyan Zhu, Huacan Wang, Jie Zhou +8LLM Agent Harnesses Personal AI Agents
Midea AIRC
From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Execution
Apr 13, 2026Hu WeiLLM Agent Orchestration LLM Agents
Resilient Write: A Six-Layer Durable Write Surface for LLM Coding Agents
Apr 12, 2026Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah +2LLM Agent Reliability AI Coding Agents
Sperixlabs, Ghana · Kwame Nkrumah University of Science and Technology, Kumasi, Ghana · VIA Cybersecurity Lab, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana
ClawVM: Harness-Managed Virtual Memory for Stateful Tool-Using LLM Agents
Apr 11, 2026Mofasshara Rafique, Laurent BindschaedlerLLM Context Management Agent Memory Management
Independent Researcher Switzerland · Max Planck Institute for Software Systems Saarbrücken, Saarland, Germany
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Apr 9, 2026Chenyu Zhou, Huacan Chai, Wenteng Chen +18LLM Agents LLM Agent Harnesses
Shanghai Jiao Tong University · Sun Yat-Sen University · Shanghai Innovation Institute +2
LogAct: Enabling Agentic Reliability via Shared Logs
Apr 9, 2026Mahesh Balakrishnan, Ashwin Bharambe, Davide Testuggine +7LLM Agent Reliability AI Agent Monitoring
Meta
MIMIC-Py: An Extensible Tool for Personality-Driven Automated Game Testing with Large Language Models
Apr 9, 2026Yifei Chen, Sarra Habchi, Lili WeiAutomated Game Testing Automated Software Testing
Electrical and Computer Engineering, McGill University, Montréal, Québec, Canada · Cohere, Montréal, Québec, Canada
How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM's Residual Role in a Planning Agent
Apr 8, 2026Sungwoo Jung, Seonil SonAutomated Agent Evaluation LLM Agent Harnesses
Independent Researcher · RLWRLD.AI Seoul, South Korea
Qualixar OS: A Universal Operating System for AI Agent Orchestration
Apr 7, 2026Varun Pratap BhardwajMulti-Agent Orchestration LLM Agent Orchestration
Independent Researcher, Solution Architect, India

May 28, 2026J/K move · Enter open · S save

Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi+13

1The Pennsylvania State University · 2UC Santa Cruz · 3Amazon+3

LLM Agent Evaluation Agent Harness Optimization LLM Agent Self-Improvement LLM Agent Harnesses Self-Evolving Agents

Open paper AskPDF arXiv

LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at https://github.com/A-EVO-Lab/a-evolve/tree/release/harness-evolution.