LLM Agents

LLM: Large Language Model

Ask about this topic

Overview

LLM agents combine language models with tools, memory, and execution environments to pursue goals through sequences of decisions rather than producing a single response. Common approaches include ReAct-style planning and tool use, retrieval and reusable skill libraries, reflection and verification, multi-agent coordination, and training with supervised trajectories or reinforcement learning; runtime harnesses also manage state, permissions, and model routing. They are tested on interactive web and coding tasks, tool-use and long-horizon benchmarks, domain workflows, and simulations of social or physical settings. A central challenge is making agents reliable and safe when tasks, instructions, and environments change while controlling cost and evaluating performance beyond narrow benchmark success.

Latest papers

CardsList

711 papers129 papers in the last four weeks, up 59% on the four weeks before. 0.7% of all new papers.

Jan 4, 2026

CaveAgent: Transforming LLMs into Stateful Runtime Operators

LLM-based agents are increasingly capable of complex task execution, yet current agentic systems remain constrained by text-centric paradigms that struggle with long-horizon tasks due to fragile multi-turn dependencies and context drift. We present CaveAgent, a framework that shifts tool use from LLM-as-Text-Generator'' to LLM-as-Runtime-Operator.'' CaveAgent introduces a dual-stream architecture that inverts the conventional paradigm: rather than treating the LLM's text context as the primary workspace with tools as auxiliary, CaveAgent elevates the persistent Python runtime as the central locus of state, with a lightweight semantic stream serving as its orchestrator. Beyond leveraging code generation to resolve interdependent sub-tasks (e.g., loops, conditionals) in a single step, CaveAgent introduces Stateful Runtime Management: it injects, manipulates, and retrieves complex Python objects (e.g., DataFrames, database connections) that persist across turns, unlike existing code-based approaches that remain text-bound. CaveAgent further provides a runtime-integrated skill management system that extends the Agent Skills open standard, enabling ecosystem interoperability through executable skill injections. This persistence mechanism serves as a high-fidelity external memory that reduces context drift in multi-turn interactions and preserves processed data for downstream applications without information loss. Evaluations show consistent improvement across challenging benchmarks, enabling CaveAgent to handle data scales that cause context overflow in both JSON-based and code-based agents. The accessible runtime state further provides programmatically verifiable feedback, enabling automated evaluation and reward signal generation without human annotation and establishing a structural foundation for future research in Reinforcement Learning with Verifiable Rewards (RLVR).

Maohao Ran, Zhenglin Wan, Cooper Lin +21

HKUST · HKBU · NUS +8

LLM Agent MemoryLLM AgentsLong-Horizon LLM AgentsLLM Context Management
AskPDF
Dec 18, 2025

Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation

LLM agents can combine individually non-revealing tool returns and disclose a sensitive conclusion, creating Tools Orchestration Privacy Risk (TOP-R). We formalize TOP-R through three conditions: conclusion sensitivity, single-source non-inferability, and compositional inferability. We introduce Library-Grounded Reverse-Inference Seed Expansion (LRSE), a four-library reverse-construction pipeline, and use it to build TOP-Bench, a 1,000-instance benchmark evaluated under a controlled two-stage tool-use protocol. Across six LLM agents, average task completion, leakage, and H-score are 98.0 percent, 88.6 percent, and 20.4. With native reasoning enabled, four models average 81.4 percent final-response leakage and 82.4 percent reasoning-trace leakage. With reasoning disabled, three prompt-only safeguards improve H-score by an average of about 3.4 points on TOP-Bench. We further propose TOP-Align, an SFT+DPO method for learning safer task-completion boundaries. On a separate post-training evaluation set, TOP-Align improves H-score by 16.2 points over the base model, versus a 5.0-point average gain from prompt-only mitigation on the same set. These results show that TOP-R requires defenses beyond prompting alone. Dataset and code are available at https://github.com/1Ponder/TOP-R.

Yuxuan Qiao, Dongqin Liu, Hongchang Yang +2

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

LLM Agent SecurityLLM AgentsLLM Safety Alignment
AskPDF
Dec 11, 2025

Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learning

Large language model (LLM) agents exhibit strong mathematical problem-solving abilities and can even solve International Mathematical Olympiad (IMO) level problems with the assistance of formal proof systems. However, due to weak heuristics for auxiliary constructions, AI for geometry problem solving remains dominated by expert models such as AlphaGeometry 2, which rely heavily on large-scale data synthesis and search for both training and evaluation. In this work, we make the first attempt to build a medalist-level LLM agent for geometry and present InternGeometry. InternGeometry overcomes the heuristic limitations in geometry by iteratively proposing propositions and auxiliary constructions, verifying them with a symbolic engine, and reflecting on the engine's feedback to guide subsequent proposals. A dynamic memory mechanism enables InternGeometry to conduct more than two hundred interactions with the symbolic engine per problem. To further accelerate learning, we introduce Complexity-Boosting Reinforcement Learning (CBRL), which gradually increases the complexity of synthesized problems across training stages. Built on InternThinker-32B, InternGeometry solves 44 of 50 IMO geometry problems (2000-2024), exceeding the average gold medalist score (40.9), using only 13K training examples, just 0.004% of the data used by AlphaGeometry 2, demonstrating the potential of LLM agents on expert-level geometry tasks. InternGeometry can also propose novel auxiliary constructions for IMO problems that do not appear in human solutions.

Haiteng Zhao, Junhao Shen, Yiming Zhang +7

1Shanghai AI Laboratory · 2Shanghai Jiao Tong University · 3Peking University +2

LLM AgentsReinforcement LearningNeuro-Symbolic Reasoning
AskPDF
Dec 10, 2025

Rethinking Query Optimization for Multi-Agent Systems [Vision]

The proliferation of large language models (LLMs) has accelerated the adoption of agent-based data pipelines. Yet current approaches remain ad hoc, relying on fixed structures, predefined LLMs, and single execution engines, without systematic optimization across heterogeneous data sources and engines. This paper presents NOMA, a query optimization framework for multi-agent data pipelines. We argue that optimizing agentic pipelines is a fundamentally different query optimization problem, with central challenges: (i)~a multi-dimensional search and objective space, where topology, model, and engine choices must be optimized jointly across latency, cost, and accuracy; (ii)~a variable pipeline topology; (iii)~the co-existence of diverse data models, leaving no common operator algebra; and (iv)~the significant cost of executing these pipelines. Our controlled experiment over a real-world 10-agent pipeline reveals extreme variance (153x cost, 5x latency, 25% quality) and that optimal plans are heterogeneous configurations no user would construct manually. Our analysis of real deployments confirms these inefficiencies are systematic. We present \system{} as an integrated optimization loop in which plan generation, cost estimation, runtime refinement, and semantic caching reinforce one another across executions, setting a community-wide research agenda on query optimization for multi-agent systems.

Zoi Kaoudi, Ioana Giurgiu

IT University of Copenhagen · IBM Research Europe

Multi-Agent System OptimizationMulti-Agent OrchestrationLLM Agent Workflow OptimizationLLM AgentsMulti-Agent Systems
AskPDF
Dec 9, 2025

Evolving Excellence: Automated Optimization of LLM-based Agents

Agentic AI systems built on large language models (LLMs) offer significant potential for automating complex workflows, from software development to customer support. However, LLM agents often underperform due to suboptimal configurations; poorly tuned prompts, tool descriptions, and parameters that typically require weeks of manual refinement. Existing optimization methods either are too complex for general use or treat components in isolation, missing critical interdependencies. We present ARTEMIS, a no-code evolutionary optimization platform that jointly optimizes agent configurations through semantically-aware genetic operators. Given only a benchmark script and natural language goals, ARTEMIS automatically discovers configurable components, extracts performance signals from execution logs, and evolves configurations without requiring architectural modifications. We evaluate ARTEMIS on four representative agent systems: the ALE Agent for competitive programming on AtCoder Heuristic Contest, achieving a \textbf{13.6%13.6\% improvement} in acceptance rate; the Mini-SWE Agent for code optimization on SWE-Perf, with a statistically significant 10.1% performance gain; and the CrewAI Agent for cost and mathematical reasoning on Math Odyssey, achieving a statistically significant \textbf{36.9%36.9\% reduction} in the number of tokens required for evaluation. We also evaluate the MathTales-Teacher Agent powered by a smaller open-source model (Qwen2.5-7B) on GSM8K primary-level mathematics problems, achieving a 22% accuracy improvement and demonstrating that ARTEMIS can optimize agents based on both commercial and local models.

Paul Brookes, Vardan Voskanyan, Rafail Giavrimis +18

TurinTech AI, London, UK · University of Surrey, Guildford, United Kingdom · University of Leeds, Leeds, UK +4

Large Language Model-Guided OptimizationLLM Agent Workflow OptimizationLLM AgentsGenetic Algorithms
AskPDF
Nov 21, 2025

REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines. These models include unimodal vision encoders and multimodal architectures. FMs are adapted to diverse perception tasks, such as image classification, change detection, and visual question answering. However, selecting the most suitable remote sensing foundation model (RSFM) for a specific task remains challenging due to scattered documentation, heterogeneous formats, and complex deployment constraints. To address this, we first introduce the RSFM Database (RS-FMD), the first structured and schema-guided resource covering over 160 RSFMs trained on various data modalities, spanning different spatial, spectral, and temporal resolutions, considering different learning paradigms. Built upon RS-FMD, we further present REMSA, a constraint-aware agent that enables automated RSFM selection from natural language queries. REMSA combines structured FM metadata retrieval with a task-driven decision workflow. In detail, it interprets user input, clarifies missing constraints, ranks models via in-context learning, and provides transparent justifications. Our system supports various RS tasks and data modalities, enabling personalized, reproducible, and efficient FM selection. To evaluate REMSA, we construct a benchmark of 100 expert-verified RS query scenarios. Each query is evaluated across 4 systems and 3 LLM backbones, with the top-3 selected models manually assessed by domain experts. This results in 3,000 expert-scored task--system--model configurations under our novel expert-centered evaluation protocol. REMSA outperforms multiple baselines, showing its practical utility in real decision-making applications. REMSA operates entirely on publicly available metadata of open source RSFMs, without accessing private or sensitive data.

Binger Chen, Tacettin Emre Bök, Behnood Rasti +2

Humboldt-Universität zu Berlin, Berlin, Germany · 5U AI, Munich, Germany · Technische Universität Berlin & BIFOLD, Berlin, Germany

Model SelectionGeospatial Foundation ModelsRemote SensingLLM Agents
AskPDF
Oct 1, 2025

Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis

We present domain-adapted large language model agents designed to support Cherenkov Telescope Array operation and data analysis. The agents combine contextual knowledge with automated validation and iterative correction to produce more reliable outputs. This approach reduces manual effort, improves consistency, and helps accelerate operational and scientific workflows. The results demonstrate the potential of agentic systems as practical assistants in specialized research environments.

Dmitriy Kostunin, Elisa Jones, Vladimir Sotnikov +3

Deutsches Elektronen-Synchrotron DESY, 15738 Zeuthen, Germany · JetBrains Limited, 8046 Paphos, Cyprus · JetBrains GmbH, 80639 München, Germany +1

High-Energy PhysicsLLM AgentsLLM Agent ReliabilityData Analysis Agents
AskPDF
Sep 25, 2025

Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion

Understanding persuasion is critical for the safety and reliability of multi-agent systems built on large language models (LLMs). This paper studies persuasion dynamics by contrasting general LLMs with Large Reasoning Models (LRMs) that employ explicit ``thinking'' processes. Through large-scale experiments on objective (MMLU) and subjective (PersuasionBench and Perspectrum) tasks, we identify Persuasion Duality: reasoning enhances an agent's persuasive power while simultaneously increasing its resistance to persuasion. For LRMs, adding thinking content increases persuasion rates by 21 pp on average, yet reduces susceptibility to incorrect persuasion by up to 10 pp on objective tasks. Despite these gains, we uncover a critical vulnerability: persuasiveness often stems from superficial cues such as response length and repetition rather than logical validity. Non-semantic padding or repeated conclusions can match or exceed the persuasive effect of coherent reasoning, revealing a strong length bias in agents' judgments. We further show that persuasion propagates non-linearly in multi-hop agent chains, where intermediate agents may amplify or attenuate influence depending on task subjectivity. Finally, guided by attention analysis, we propose a prompt-level adversarial argument detection method that consistently improves agent robustness.

Haodong Zhao, Jidong Li, Zhaomin Wu +4

School of Computer Science, Shanghai Jiao Tong University · 2National University of Singapore · 3Inner Mongolia Research Institute, Shanghai Jiao Tong University

LLM AgentsPersuasion in Language ModelsLLM Agent ReliabilityLarge Reasoning Models
AskPDF
Sep 3, 2025

Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation

LLM agents are increasingly proposed as substitutes for human participants in social research, usually on the strength of survey responses that match those of the humans they represent. We argue that such output-level alignment is not enough. Drawing on the behavioral sciences, we propose Latent Profiling, a framework for testing behavioral coherence: whether an agent's conversational behavior follows, as human behavioral models predict, from its own latent profile elicited by short surveys. As a case study, we test six hypotheses about preference, openness and agreement across nine open models. Agents reproduce surface-level trends but systematically fail in-depth tests: they are not behaviorally coherent, even when their individual responses look human. Latent Profiling thus offers a theory-grounded instrument for a behavioral science of LLMs.

James Mooney, Josef Woldense, Zheng Robert Jia +4

Department of Computer Science and Engineering, University of Minnesota · Department of African American & African Studies, University of Minnesota · Department of Sociology, University of Chicago +1

Behavioral Consistency of LLM AgentsLLM Agent EvaluationLLM AgentsLarge Language Model-Based Social SimulationHuman Behavior Simulation
AskPDF
Aug 25, 2025

LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios

Recent advances in LLM-based agents highlight the importance of their reasoning frameworks, which guide the problem-solving process in diverse ways. This survey introduces a unified formal language to systematically categorize these frameworks at three compositional levels: single-agent, tool-based, and multi-agent methods. Following our taxonomy, we review key application scenarios across scientific discovery, healthcare, software engineering, society, economics, and general-purpose tasks. It also compares the distinct features and evaluation strategies of each category. Through our taxonomy and comparisons, our survey explores the designs and strengths of LLM-based agentic frameworks in different scenarios, reviewing the fast-paced development of complex agentic systems in the real world.

Bingxi Zhao, Lin Geng Foo, Ping Hu +3

Multi-Agent LLM SystemsLLM Agents
AskPDF
Aug 17, 2025

LinkAnchor: An Autonomous LLM-Based Agent for Issue-to-Commit Link Recovery

Issue-to-commit link recovery in software repositories is fundamental to software traceability and project management, yet it remains a challenging task. Prior studies show that only about 42.2% of issues on GitHub are correctly linked to their commits, highlighting the need for more effective solutions. Existing work has explored a range of ML/DL approaches, and more recently, large language models (LLMs) have been applied to this problem. However, these methods face two major limitations. First, LLMs are restricted by limited context windows and cannot simultaneously process all available data sources, such as long commit histories, extensive issue discussions, and large code repositories. Second, most approaches operate on individual issue-commit pairs, where a model independently scores the relevance of a single commit to an issue. This pairwise formulation fails to account for the complex associativity of software fixes, where an issue is often resolved by an aggregate chain of commits rather than a single atomic change. By ignoring these temporal and parental dependencies, existing methods often fail to incorporate the complete resolution logic and might misidentify intermediate commits as final fixes. Furthermore, this strategy is computationally inefficient in large repositories, as it requires exhaustively evaluating an enormous number of candidate pairs. To address these challenges, we present LinkAnchor, the first autonomous LLM-based agent designed specifically for issue-to-commit link recovery. LinkAnchor introduces a lazy-access architecture that allows the underlying LLM to dynamically retrieve only the most relevant contextual data, such as commits, issue comments, and code files, without exceeding token limits.

Arshia Akhavan, Alireza Hoseinpour, Abbas Heydarnoori +2

Bowling Green State University, Department of Computer Science, Bowling Green, USA · University of Nebraska-Lincoln, School of Computing, Lincoln, USA · University of Zurich, Department of Informatics, Zurich, Switzerland

Software EngineeringLLM Agents
AskPDF
Aug 5, 2025

InqEduAgent: Adaptive AI Learning Partners with Gaussian Process Augmentation

Collaborative partnerships play a crucial role in inquiry-oriented education. However, most learning partners are currently assigned through experience-driven heuristics or rule-based machine assistants, which often result in limited knowledge expansion and low adaptability. To address these challenges, this study introduces InqEduAgent, an LLM-empowered generative agent framework designed to simulate and select adaptive learning partners for inquiry-based learning. InqEduAgent integrates a Gaussian process-augmented matching mechanism to model the cognitive and evaluative characteristics of learners, allowing adaptive partner selection based on prior knowledge patterns. Comprehensive experiments demonstrate that InqEduAgent consistently achieves superior performance across diverse learning scenarios and large language model configurations. This study advances human-AI collaborative learning by enabling intelligent pairing between human- and AI-based learning partners, and contributes to adaptive user modeling and personalized recommendation within Web-based educational environments.

Wen-Xi Yang, Tian-Fang Zhao, Guan Liu

Guangdong Institute of Smart Education, Jinan University, Guangzhou, China · School of Journalism and Communication, Jinan University, Guangzhou, China

AI in EducationGenerative AI in EducationLLM AgentsAdaptive LearningHuman-AI Collaboration
AskPDF
Jul 21, 2025

Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor

Empowered by large language models (LLMs), intelligent agents have become a popular paradigm for interacting with open environments to facilitate AI deployment. However, hallucinations generated by LLMs-where outputs are inconsistent with facts-pose a significant challenge, undermining the credibility of intelligent agents. Only if hallucinations can be mitigated, the intelligent agents can be used in real-world without any catastrophic risk. Therefore, effective detection and mitigation of hallucinations are crucial to ensure the dependability of agents. Unfortunately, the related approaches either depend on white-box access to LLMs or fail to accurately identify hallucinations. To address the challenge posed by hallucinations of intelligent agents, we present HalMit, a novel black-box watchdog framework that models the generalization bound of LLM-empowered agents and thus detect hallucinations without requiring internal knowledge of the LLM's architecture. Specifically, a probabilistic fractal sampling technique is proposed to generate a sufficient number of queries to trigger the incredible responses in parallel, efficiently identifying the generalization bound of the target agent. Experimental evaluations demonstrate that HalMit significantly outperforms existing approaches in hallucination monitoring. Its black-box nature and superior performance make HalMit a promising solution for enhancing the dependability of LLM-powered systems.

Siyuan Liu, Wenjing Liu, Zhiwei Xu +3

College of Computer Science, Nankai University · College of intelligent Science and Technology, Inner Mongolia University of Technology · Institute of Computing Technology, Chinese Academy of Sciences +3

LLM Hallucination MitigationHallucination in Language ModelsAI Agent MonitoringLLM Hallucination DetectionLLM AgentsLLM Agent Reliability
AskPDF
Jun 12, 2025

AutoMind: Adaptive Knowledgeable Agent for Automated Data Science

Large Language Model (LLM) agents have shown great potential in addressing real-world data science problems. LLM-driven data science agents promise to automate the entire machine learning pipeline, yet their real-world effectiveness remains limited. Existing frameworks depend on rigid, pre-defined workflows and inflexible coding strategies; consequently, they excel only on relatively simple, classical problems and fail to capture the empirical expertise that human practitioners bring to complex, innovative tasks. In this work, we introduce AutoMind, an adaptive, knowledgeable LLM-agent framework that overcomes these deficiencies through three key advances: (1) a curated expert knowledge base that grounds the agent in domain expert knowledge, (2) an agentic knowledgeable tree search algorithm that strategically explores possible solutions, and (3) a self-adaptive coding strategy that dynamically tailors code generation to task complexity. Evaluations on two automated data science benchmarks demonstrate that AutoMind delivers superior performance versus state-of-the-art baselines. Additional analyses confirm favorable effectiveness, efficiency, and qualitative solution quality, highlighting AutoMind as an efficient and robust step toward fully automated data science. Code is at https://github.com/zjunlp/AutoMind.

Yujie Luo, Yixin Ou, Jingsheng Zheng +9

Zhejiang University · Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph · Ant Group

Data Analysis AgentsLLM AgentsAutomated Machine LearningLLM Agent Workflow Optimization
AskPDF
Jun 2, 2025

Small Language Models are the Future of Agentic AI

Large language models (LLMs) are often praised for exhibiting near-human performance on a wide range of tasks and valued for their ability to hold a general conversation. The rise of agentic AI systems is, however, ushering in a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation. Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice. We discuss the potential barriers for the adoption of SLMs in agentic systems and outline a general LLM-to-SLM agent conversion algorithm. Our position, formulated as a value statement, highlights the significance of the operational and economic impact even a partial shift from LLMs to SLMs is to have on the AI agent industry. We aim to stimulate the discussion on the effective use of AI resources and hope to advance the efforts to lower the costs of AI of the present day. Calling for both contributions to and critique of our position, we commit to publishing all such correspondence at https://research.nvidia.com/labs/lpr/slm-agents.

Peter Belcak, Greg Heinrich, Shizhe Diao +5

NVIDIA Research · Georgia Institute of Technology

Small Language ModelsLLM Agents
AskPDF
May 27, 2025

Agent-Environment Alignment via Automated Interface Generation

Large language model (LLM) agents have shown impressive reasoning capabilities in interactive decision-making tasks. These agents interact with environment through intermediate interfaces, such as predefined action spaces and interaction rules, which mediate the perception and action. However, mismatches often happen between the internal expectations of the agent regarding the influence of its issued actions and the actual state transitions in the environment, a phenomenon referred to as agent-environment misalignment. While prior work has invested substantially in improving agent strategies and environment design, the critical role of the interface still remains underexplored. In this work, we empirically demonstrate that agent-environment misalignment poses a significant bottleneck to agent performance. To mitigate this issue, we propose ALIGN, an Auto-Aligned Interface Generation framework that alleviates the misalignment by enriching the interface. Specifically, the ALIGN-generated interface enhances both the static information of the environment and the step-wise observations returned to the agent. Implemented as a lightweight wrapper, this interface achieves the alignment without modifying either the agent logic or the environment code. Experiments across multiple domains including embodied tasks, web navigation and tool-use, show consistent performance improvements, with up to a 45.67% success rate improvement observed in ALFWorld. Meanwhile, ALIGN-generated interface can generalize across different agent architectures and LLM backbones without interface regeneration. Code and experimental results are available at https://github.com/THUNLP-MT/ALIGN.

Kaiming Liu, Xuanyu Lei, Ziyue Wang +2

College of AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China

LLM AgentsAI AlignmentWeb Agents
AskPDF
Apr 16, 2025

Towards LLM Agents for Earth Observation

Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. In this work we ask: Are AI systems ready for reliable Earth Observation? To answer this, we introduce UnivEARTH, a coding benchmark of 408 yes/no questions from NASA Earth Observatory articles across 7 various topics and over 15 satellite instruments and sources. Using Google Earth Engine API as a tool in a zero-shot setup, LLM agents achieve an accuracy of 40.0% where the code fails to run over 44% of the time. To better understand LLM agent behavior, we also analyze the impact of using the JavaScript API versus Python and the effect of providing documentation. Furthermore, we find that using a Reflexion framework significantly reduces errors: Claude-4.5-Sonnet, Gemini-2.5-Pro, and GPT-5 accuracies rise to around 60%. However, these results remain only marginally above random chance. Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward.

Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10

Cornell University · Columbia University

Code GenerationLLM Agent EvaluationRemote SensingAI Agent BenchmarksLLM AgentsEarth Science
AskPDF
Feb 25, 2025

AgentRM: Enhancing Agent Generalization with Reward Modeling

Existing LLM-based agents have achieved strong performance on held-in tasks, but their generalizability to unseen tasks remains poor. Hence, some recent work focus on fine-tuning the policy model with more diverse tasks to improve the generalizability. In this work, we find that finetuning a reward model to guide the policy model is more robust than directly finetuning the policy model. Based on this finding, we propose AgentRM, a generalizable reward model, to guide the policy model for effective test-time search. We comprehensively investigate three approaches to construct the reward model, including explicit reward modeling, implicit reward modeling and LLM-as-a-judge. We then use AgentRM to guide the answer generation with Best-of-N sampling and step-level beam search. On four types of nine agent tasks, AgentRM enhances the base policy model by 8.88.8 points on average, surpassing the top general agent by 4.04.0. Moreover, it demonstrates weak-to-strong generalization, yielding greater improvement of 12.612.6 on LLaMA-3-70B policy model. As for the specializability, AgentRM can also boost a finetuned policy model and outperform the top specialized agent by 11.411.4 on three held-in tasks. Further analysis verifies its effectiveness in test-time scaling. Codes will be released to facilitate the research in this area.

Yu Xia, Jingru Fan, Weize Chen +7

Department of Computer Science and Technology, Tsinghua University · Gaoling School of Artificial Intelligence, Renmin University of China

Reward ModelingLLM Agents
AskPDF
Dec 12, 2024

How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?

Radiology, with its heterogeneous modalities, anatomies, and evolving protocols, is a natural yet high-stakes testbed for agentic AI. As LLMs grow more capable and tool ecosystems such as MCP and Agent Skills more complex, their capability boundaries in radiology remain unclear. We introduce RadA-BenchPlat, a two-layer benchmark spanning idealized reasoning and real-world execution: a synthetic layer with 2.2k clinician-verified records, 24.2k QA pairs, and 10 tool categories under diverse availability settings; and a real-environment layer pairing 165 2D/3D cases with executable tools for grounding, diagnosis, and report generation. We find that high completion rates under idealized settings do not carry over to real environments, where the best agents trail their upper bound by approximately 15% and drop to 0.327 on long-chain tasks. Prompting strategies (few-shot exemplars, multi-agent coordination) and AutoTB (on-the-fly synthesis of missing tools) mitigate execution drift and breakdowns in difficult scenarios, lifting execution success to 94.5%, yet remain not fully reliable. Overall, our findings suggest that radiology agents are no longer limited to conceptual demonstrations, but are beginning to show potential for real-world applications. Project repository: https://github.com/MAGIC-AI4Med/RadABench

Qiaoyu Zheng, Chaoyi Wu, Weike Zhao +5

Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Renmin Hospital of Wuhan University, Wuhan, China

HealthcareLLM Agent EvaluationAI Agent BenchmarksLLM AgentsLong-Horizon LLM Agents
AskPDF
Nov 15, 2024

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes. Such models are typically outcome-specific, however, requiring training data for each target outcome, limiting their applicability to new domains. We test whether large language models (LLMs) can relax these requirements by using self-report data to build attitudinal and behavioral simulations, or "generative agents," that can predict responses across outcomes without outcome-specific training data. Using data from a diverse national sample of 1,052 Americans, we built agents from (i) two-hour, semi-structured interviews elicited using the American Voices Project interview schedule, (ii) structured surveys including General Social Survey items and the Big Five personality inventory, or (iii) both sources combined. On held-out General Social Survey items, interview-only, survey-only, and combined agents achieved accuracies equal to 83%, 82%, and 86% of participants' own two-week test-retest consistency benchmark, respectively, compared with 74% for demographics-only agents. Combining interviews and surveys produced the highest accuracy, though gains over either source alone were modest, suggesting that predictive benefits from data begin to asymptote once the model has observed sufficient evidence within a domain. We find that these agents also predict personality traits, economic-game behavior, and experimental responses, while reducing accuracy disparities across racial and ideological groups relative to demographics-only agents. Together, these results show that LLM agents grounded in qualitative or quantitative self-reports can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data.

Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst +8

1Computer Science Department, Stanford University; Stanford, CA, 94305, USA. · Department of Communication Studies, Northwestern University; Evanston, IL 60208, USA. · 7CDSP and CEE, Sciences Po; Paris, 75007, France. +4

Personality Modeling in Language ModelsHuman Behavior SimulationLLM AgentsHuman Behavior PredictionLarge Language Model-Based Social Simulation
AskPDF
Aug 6, 2024

A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model

The rapid advancement of AI technology has led to widespread applications of agent systems across various domains. However, the need for detailed architecture design poses significant challenges in designing and operating these systems. This paper introduces a taxonomy focused on the architectures of foundation-model-based agents, addressing critical aspects such as functional capabilities and non-functional qualities. We also discuss the operations involved in both design-time and run-time phases, providing a comprehensive view of architectural design and operational characteristics. By unifying and detailing these classifications, our taxonomy aims to improve the design of foundation-model-based agents. Additionally, the paper establishes a decision model that guides critical design and runtime decisions, offering a structured approach to enhance the development of foundation-model-based agents. Our contributions include providing a structured architecture design option and guiding the development process of foundation-model-based agents, thereby addressing current fragmentation in the field.

Jingwen Zhou, Qinghua Lu, Jieshan Chen +4

School of Computer, Data and Mathematical Sciences, Western Sydney University, Sydney, Australia · CSIRO’s Data61, Australia · University of New South Wales, Sydney, New South Wales, Australia +1

LLM AgentsAgentic AI
AskPDF