LLM Agent Harnesses

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

49 papers in the last four weeks, up 88% on the four weeks before. 0.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 198

All topics

Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent
Jul 19, 2026Arunabh DastidarAI Agent Reliability LLM Answer Verification
for the Leni Team · Leni Inc.
DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
Jul 18, 2026Runming He, Zhen Hao Wong, Hao Liang +4LLM Grounding Code Generation
Peking University · Institute for Advanced Algorithms Research, Shanghai · Zhongguancun Academy
Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents
Jul 17, 2026Zhengyu Chen, Teng Xiao, Huaisheng Zhu +3Agent Harness Optimization LLM Agent Workflow Optimization
Meituan · Allen Institute for AI · Independent
Recursive Harness Self-Improvement
Jul 17, 2026Hyunin Lee, Jinglue Xu, Jeffrey Seely +3Continual Learning for LLM Agents Agent Harness Optimization
1Sakana AI, 2UC Berkeley
Harnessing LLMs for Reliable Academic Supervision: A Comparative Study
Jul 16, 2026Akash RajLLM Reliability LLM Agent Harnesses
M.Tech, School of Artificial Intelligence and Data Science Indian Institute of Technology Jodhpur
RetroAgent: Harnessing LLMs to Search Over Structured Memory for Agentic Retrosynthesis Planning
Jul 16, 2026Yanqiao Zhu, Jingru Gan, Xiaoqi Sun +8LLM Agent Harnesses LLM Agent Planning
UCLA · MIT · Amazon +1
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
Jul 15, 2026Kai Chen, Zichen Ding, Jiaye Ge +20LLM Agent Evaluation AI Agent Benchmarks
AgentCompass Team, Shanghai AI Laboratory
Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity
Jul 15, 2026Xiaotian Luo, Fengxingyu Wang, Chuanrui Hu +2LLM Agent Self-Improvement Quality-Diversity Optimization
EverMind AI · Shanda Group
Structured Feedback Improves Repair in an LLM Agent Loop
Jul 15, 2026Jaideep Ray, Ankit GoyalLLM Self-Correction LLM Agent Evaluation
Independent Researcher
MyAG: A Graph-Based Framework for Designing and Analyzing Composable LLM Agent Systems
Jul 15, 2026Zhisong ZhangLLM Agent Orchestration LLM Agents
Department of Computer Science, City University of Hong Kong
Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
Jul 14, 2026Ruhan Wang, Yucheng Shi, Zongxia Li +7Software Engineering Agents LLM Agent Harnesses
Tencent · Indiana University · University of Maryland, College Park +2
PalmClaw: A Native On-Device Agent Framework for Mobile Phones
Jul 14, 2026Hongru Cai, Yongqi Li, Ran Wei +1Tool-Using Agents LLM Agents
The Hong Kong Polytechnic University · Hangzhou Diagens Biotechnology Co., Ltd.
Rethinking the Evaluation of Harness Evolution for Agents
Jul 14, 2026Yike Wang, Huaisheng Zhu, Zhengyu Hu +8LLM Agent Evaluation Agent Harness Optimization
Allen Institute for AI · University of Washington · Independent
ToFu: A White-Box, Token-Efficient Agent Harness for Researchers
Jul 13, 2026Junhao Ruan, Yuan Ge, Bei Li +7Tool-Augmented Language Model Agents Software Engineering Agents
Northeastern University, China · LongCat RSI, Meituan · NiuTrans Research
Agentic Routing: The Harness-Native Data Flywheel
Jul 13, 2026Xinchen Liu, Hang Zhou, Yingjie Zong +12LLM Agent Training LLM Agent Harnesses
TokenRhythm Technologies
StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure
Jul 13, 2026Wenyi Wu, Sibo Zhu, Kun Zhou +3Computer-Use Agents Long-Horizon LLM Agents
University of California, San Diego · 2Aether AI Lab · §Work done during an internship at Aether AI.
Compile, Then Page: Executable SOP Programs and a Capability-Gated Runtime for Procedural LLM Agents
Jul 13, 2026Chenglin Yu, Li Yin, Ying Yu +4Instruction Following Runtime Enforcement for AI Agents
Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University · Department of Data and Systems Engineering, The University of Hong Kong · College of Economics and Management, Zhejiang Normal University +3
A Formal Hierarchical Architecture for Agentic Orchestration with Stack-Based Execution and Lazy Discovery
Jul 13, 2026Prashant Devadiga, Abhishek, Adithya Mishra +9LLM Agent Orchestration Tool-Using Language Agents
UPI Help
LOGOS: A Living Logic for AI Agent Teams That Evolve With Humans
Jul 12, 2026Yuma Ichikawa, Yamato Arai, Kosaku Kimura +2Multi-Agent LLM Systems Human-in-the-Loop AI
1Fujitsu Limited · 2RIKEN Center for AIP · 4Tokai University
Compaction as Epistemic Failure: How Agentic LLM Tools Fabricate Confirmed Results from Killed Processes
Jul 11, 2026Hiroki TambaAI Agent Reliability LLM Agent Memory
Independent Researcher
MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
Jul 11, 2026Chengguang Gan, Hanjun Wei, Yunhao Liang +3Computer-Use Agents Web Agent Benchmarks
Independent Researcher · University of Chinese Academy of Sciences · Monash University +2
Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents
Jul 10, 2026Izumi Takahara, Teruyasu MizoguchiAI Agents for Scientific Discovery LLM Auditing
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan 153-8505
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks
Jul 9, 2026Zhekai Chen, Chengqi Duan, Kaiyue Sun +4AI Agent Benchmarks Multimodal Agents
HKU MMLab · Meituan
TTHE: Test-Time Harness Evolution
Jul 9, 2026Jun Nie, Yonggang Zhang, Jun Song +5Test-Time Adaptation Agent Harness Optimization
Hong Kong Baptist University · University of Science and Technology of China · Hong Kong Generative AI Research & Development Center, The Hong Kong University of Science and Technology +1
From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Jul 9, 2026Joongho Ahn, Moonsoo KimLLM Auditing LLM Guardrails
AI Leadership Research Center
The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI
Jul 8, 2026Muayad Sayed Ali, Aliaksandra Novik, Anji Boddupally +28LLM Inference Efficiency LLM Agent Orchestration
Writer, Inc.
Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1
Jul 7, 2026Kabir Moghe, Peter ChinLLM-Based Program Synthesis Agentic Reasoning
Department of Computer Science Dartmouth College
Harnessing Code Agents for Automatic Software Verification
Jul 7, 2026Shuangxiang Kan, Shuanglong Kan, Sebastian ErtelSoftware Engineering Agents Automated Theorem Proving
Singapore Management University · Singapore · Barkhausen Institut +1
Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents
Jul 5, 2026Haiwen Yi, Xinyuan SongSoftware Engineering Agents LLM Agent Evaluation
University of Toronto · Emory University
Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning
Jul 5, 2026Haiwen Yi, Xinyuan SongOffline RL LLM Agent Harnesses
University of Toronto · Emory University

Jul 19, 2026, cs.SEJ/K move · Enter open · S save

Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

Arunabh Dastidar

for the Leni Team · Leni Inc.

AI Agent Reliability LLM Answer Verification LLM Agent Evaluation LLM Agent Verification AI Agent Benchmarks LLM Agent Harnesses+1

Open paper AskPDF arXiv

Multi-step enterprise agent tasks fail in a characteristic way: single-pass inference has no checkpoint between deciding an answer and committing to it. We study one production system (Leni) whose architecture installs such checkpoints: verification loops (execute, observe, compare, correct) staffed by lightweight task-specialized post-trained models. We evaluate the unmodified production configuration on three public benchmarks stressing distinct failure modes: SpreadsheetBench Verified (silent computation error), BullshitBench v2 (premise confabulation), and the GAIA validation split (cascade error over long tool chains). The full system improves over its frontier base model by +11.0 percentage points on SpreadsheetBench (91.25% vs 80.25%, n=400, p<0.001), +7 to +10 percentage points on BullshitBench (98% vs 91%, n=100), and roughly +15 points on GAIA validation (75.2% pass@1, n=165; 83.0% best-of-k). Our central contribution is a decomposition of that uplift: most of it comes from scaffolding, routing, and specialist models rather than from the verification step itself, whose isolated contribution is small (+1.5 points) but concentrated at the top of the score distribution, where it converts otherwise-failing tasks. We instrument the loop end-to-end, yielding an empirical verifier confusion matrix (catch rate about 0.20, fix rate 0.75, no false-alarm regressions) that grounds a compounding-reliability model. Specialist-swap ablations suggest that the loop's value depends on who observes it: replacing the small trained verifier with the generating frontier model eliminates most rescues. A valid-premise control shows zero over-rejections in 100 expert-level questions.

Momentum

Related topics

Latest papers 198