A Graph-Based Reinforcement Learning Framework for Structured Drift Diagnosis and Recovery in Autonomous LLM Agents

Ismail El Hamraoui; Sagar Jose; Nicolas Bureau; Robert Plana

cs.AIAug 14, 2026

A Graph-Based Reinforcement Learning Framework for Structured Drift Diagnosis and Recovery in Autonomous LLM Agents

Authors: Ismail El Hamraoui, Sagar Jose, Nicolas Bureau, Robert Plana

Organizations: Digital Excellence Center, Assystem EOS, France

Abstract

Autonomous LLM agents are increasingly deployed in complex real-world workflows, yet they remain vulnerable to runtime behavioral drift, a silent deviation from the original task that can lead to irreversible side effects on external systems. Existing approaches address drift at the prompt level but lack structured mechanisms for step-level detection, risk assessment, and recovery decision. Because the main task-executing agent is often a large and expensive model that cannot be re-trained on every deployment, this work targets a plug-and-play recovery module instead. It introduces a graph-based framework in which a single small language model is trained via reinforcement learning to specialize at each node of a recovery graph, external to the main agent. Each node has a precise role,: drift classification, operation detection, risk evaluation, or final decision and the model learns to produce structured XML-formatted reasoning adapted to that role. Training combines rule-based structural rewards with an LLM-as-judge semantic-quality signal, so that the model is graded both on how it answers (schema and length) and on what it says. Experiments on the public AppWorld benchmark show that the method generally exploits information about the suspected drift onset to issue correct recovery decisions using a small language model. In addition, the trained small language model reliably respects the prescribed output schema and produces semantically appropriate content in each field according to its assigned node role.

Figures & tables

Figure 1: Desiderata for a drift-recovery framework in a deployed LLM-agent setting.

Figure 2: Training pipeline. The five-node recovery graph (left) emits a node-tagged prompt $q$ at each audited step. The policy $\pi_{\theta}$ samples $G$ completions per prompt in the shared $\langle\texttt{reasoning}\rangle+\langle\texttt{answer}\rangle$ XML schema , each scored by the composite reward $R(o,\nu,q)=\sum_{k=1}^{9}w_{k}\,r_{k}(o,\nu,q)$ that combines eight structural checks with a frozen LLM-as-judge signal . Group-normalized advantages $\hat{A}_{i}$ drive the GRPO update, specializing $\pi_{\theta}$ into all five node roles.

	Granite 3.3 2B		Qwen 2.5 1.5B
	base	our method	base	our method
$R$ (mean)	3.68	5.15	0.56	4.80
$r_{\text{judge}}$	0.71	0.90	0.47	0.66

Table 1: Held-out intrinsic evaluation on 100 node prompts (

20

per node).

R

is the composite reward (Eq. 15 );

r_{\text{judge}}

is the LLM-as-judge sub-score.

Panel of Figure 3 — Figure 3: Per-prompt gain $\Delta R=R_{\text{trained}}-R_{\text{base}}$ on the 100 held-out prompts. The red dashed line marks $\Delta R=0$ : every prompt is to its right, for both models.

Panel of Figure 4 — Figure 4: Composite reward by node.

Panel of Figure 5 — Figure 5: LLM-as-judge sub-score $r_{\text{judge}}$ by node.

Setting	D1	D2	D3	Agg.
No drift	71.4	7.7	23.1	35.3
Drift, no recovery	0.0	0.0	0.0	0.0
Granite base	0.0	0.0	0.0	0.0
GPT-4o recovery	57.1	15.4	23.1	32.5
Granite trained	57.1	7.7	15.4	27.5

Table 2: End-to-end recovery under Type I drift with a GPT-4o initial agent. Values are task_goal_completion (%) on 40 AppWorld tasks.

Setting	D1	D2	Agg.
No drift	50.0	7.7	20.0
Drift, no recovery	0.0	0.0	0.0
Granite trained	21.5	0.0	7.5
GPT-4o-mini recovery	35.7	0.0	12.5

Table 3: End-to-end recovery under Type II drift with a GPT-4o-mini initial agent.

Setting	D1	D2	D3	Agg.
No drift	78.6	15.4	30.8	42.5
Drift, no recovery	0.0	0.0	0.0	0.0
Granite trained	85.7	0.0	14.4	35.0
GPT-4o recovery	85.7	15.4	30.8	45.0

Table 4: End-to-end recovery under Type II drift with a GPT-4o initial agent.

Panel of Figure 6 — Figure 6: Per-component reward decomposition.

Panel of Figure 7 — Figure 7: Composite reward per held-out prompt (base vs. our method). The trained curve dominates the base curve everywhere.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Group	Hyperparameter	Value
GRPO	Group size $G$	$4$
	Clip $\epsilon$	$0.2$
	KL coefficient $\beta$	$0.04$
	Max gradient norm	$0.1$
Sampling	Temperature	$0.9$
	top- $p$	$1.0$

Appendix

Table 5: training configuration.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 838 papers

When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents
Oct 27, 2026Chiming Duan, Tong Jia, Yingxi Chen +3LLM Agents LLM Agent Reliability
Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China
Experience Memory Graph: Extracting Reusable Correction Knowledge from Agent Trajectories
Jul 15, 2026Wenjun Wang, Yuchen Fang, Fengrui Liu +2Agent Memory LLM Self-Correction
University of Electronic Science and Technology of China, Chengdu, China · Yangtze Delta Region Institute (Quzhou), School of Computer Science and Engineering, UESTC · Shenzhen Institute for Advanced Study, UESTC
Robust Function Calling for LLM Agents Under Interface Drift
Oct 16, 2026Aniruddha Chattopadhyay, Kanishka Halder, Partha Sarathi RoyTool-Augmented Language Model Agents LLM Agent Reliability
Independent Researcher

Oct 27, 2026J/K move · Enter open · S save

When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents

Chiming Duan, Tong Jia, Yingxi Chen, Minghua He+2

Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China

LLM Agents LLM Agent Reliability LLM Agent Evaluation

Open paper AskPDF arXiv

LLM agents interact with Web services to perform tasks on users' behalf. Errors during these interactions can disrupt execution and prevent task completion, motivating external intervention. However, agents may self-recover through continued execution, raising the question of when to intervene and when to wait for self-recovery. We present an empirical study of errors and self-recovery in LLM agents. We construct ReInBench, a dataset containing 1,259 agent traces and 1,403 error events across computer-use and Web tasks. Our study examines how often and how quickly agents self-recover, how self-recovery relates to subsequent execution and task completion, and whether self-recovery can be predicted when an error occurs. We find that 725 error events self-recover, often within a few steps, with self-recovery performance differing across Execution LLMs. Self-recovery can also help agents recover from subsequent errors of the same type and support overall task completion. Intervention should therefore prioritize errors unlikely to self-recover, while allowing agents to continue when self-recovery is likely. Building on these empirical findings, we propose ReCoF, a method that predicts whether an error will self-recover to inform whether additional intervention is needed. ReCoF achieves an average relative improvement of 9.02% in Macro-F1 over the strongest baseline on ReInBench. Our code and data can be found at https://method-notes.github.io/ReCoF/.

Abstract

LLM Agent Reliability

Explore similar work

When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents

Experience Memory Graph: Extracting Reusable Correction Knowledge from Agent Trajectories

Robust Function Calling for LLM Agents Under Interface Drift