SynAct: Synergistic Active Feedback Learning for LLM Agents

Ning Liu; Ian Li; Kristina Klinkner

cs.LGOct 8, 2026

SynAct: Synergistic Active Feedback Learning for LLM Agents

Authors: Ning Liu, Ian Li, Kristina Klinkner

Organizations: Amazon

Abstract

LLM agents executing multi-step workflows fail in two fundamentally different ways: recoverable slips, which further attempts can repair, and knowledge gaps, which retrying cannot resolve. Expert feedback can address the latter, but expert attention is scarce. The key challenge is therefore not whether to ask for feedback, but where feedback can change the outcome. Existing approaches either avoid expert feedback, allowing agents to reinforce their own errors, or allocate feedback at the task level without identifying consequential decisions. We present SynAct, an active learning framework that treats expert attention as the binding constraint and allocates feedback at the level of individual decisions. SynAct ranks decisions by the mutual information between the selected action and the terminal outcome across repeated rollouts, queries an expert at the highest-value decisions, and distills the resulting corrections into transferable rules that the agent can reuse without expert access at deployment. We theoretically characterize what this signal identifies under finite repeated sampling: with k rollouts and at least m observations per action, outcome-constant decisions have zero mutual information, while outcome-separating decisions have mutual information at least H2(m/k) bits, yielding exact separation for any threshold t in (0,H2(m/k)). This also explains why success-based acquisition can be blind when all rollouts fail: success is constant even when failed runs reach different outcomes. Across four agent benchmarks and four models spanning three families, SynAct improves held-out accuracy in all 16 evaluated settings, by +11.7pp on average. The same signal transfers to inference-time routing, where SynAct outperforms budget-matched cascade, uncertainty, and expert-judge routers in 15 of 16 settings, while a single targeted correction outperforms seven additional unaimed retries.

Figures & tables

Figure 1: SynAct allocates a fixed expert budget over decision points rather than episodes. Acquisition scores each state-keyed decision node by $I(A;O)$ , computed from the run itself with no gold answer or success check. Authoring sends the top-scoring nodes to the expert in one call, which abstracts them into a convention plus an antipattern. Application injects the convention only into failed attempts exhibiting that antipattern, iteration re-rolls to surface residual failures, and the same score routes consultation per task at inference time.

Figure 2: Training-time methods with an equal expert budget (agent Qwen3-32B, one panel per benchmark): ACE, ReasoningBank, AWM, AWM-iter, and SynAct at its best iteration, all given the same expert. Table 5 reports all four agent models.

Figure 3: Gain vs. cumulative expert tokens (TextCraft, Qwen3-32B, log

x

): SynAct over ten iterations, distillers at their measured budgets.

Figure 4: Inference-time routing at a matched expert budget (

\theta{=}0.6

R{=}3

, one panel per benchmark): self-reflection with ten retries, a token-matched cascade, SynAct, and SynAct with the iteration-10 rule active. Table 9 adds the action-entropy and expert-judge routers.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$p_{v,a}$	0.10	0.20	0.25	0.30	0.40	0.50
$\Pr[N_{v,a}\geq 2]$	0.187	0.497	0.633	0.745	0.894	0.965

Appendix

Table 1: Arm recovery at the default budget

k{=}8

m{=}2

, from Eq. 6 . An arm the agent takes in half its rollouts is retained with probability

0.965

, whereas one taken a tenth of the time is retained with probability

0.187

. The support floor therefore selects reproducible decision components and discards singleton behavior.

	$k$
Quantity ( $m{=}2$ , $c{=}2$ )	4	6	8	10
$P_{\text{sc}}$	0.375	0.781	0.930	0.979
$\mathbb{E}[\mathcal{I}(v)\mid\text{scorable}]$	0.281	0.177	0.124	0.094
$\Pr[\mathcal{I}(v)\geq 0.3\mid\text{scorable}]$	0.625	0.219	0.159	0.063
$\mathbb{E}[\mathcal{I}(v)\mid\text{scorable}]$ , $c{=}3$	0.463	0.328	0.247	0.195
$\mathbb{E}[\mathcal{I}(v)\mid\text{scorable}]$ , $c{=}4$	0.574	0.435	0.346	0.284

Appendix

Table 2: Exact null behavior of the plug-in score when

O

is independent of

A_{v}

, so the true mutual information is

0

. Computed by complete enumeration over all realizations, conditional on the node being scorable.

P_{\text{sc}}

is the probability of scorability. The estimator is upward-biased, and the bias grows with the number of outcome classes

c

, which is why we rank by

\mathcal{I}(v)

rather than threshold it at

0

Benchmark	$n_{\text{train}}$	$n_{\text{test}}$	Steps	Workers	Split rule
InterCode-SQL	50	150	10	8	deterministic shuffle, first 50 train, rest test
TextCraft	50	59	20	8	depth-3 recipe seeds shuffled, split in half
InterCode-Bash	20	40	10	1	fs_1 shuffled, first 20 train, rest test
$\tau$ -bench retail	50	115	20	4	benchmark’s own train/test task split

Appendix

Table 3: Per-benchmark configuration. Steps is the hard cap on agent actions per rollout. Workers is the concurrency cap used during evaluation, which is 1 for the Docker-backed shell environment because that environment is not thread-safe.

		Rollouts $k$				Budget $B$
Bench	Model	base	2	4	6	base	10%	25%	50%
SQL	Q	65.5	69.3	69.6	70.4	65.9	70.9	71.1	66.7
	M	51.6	63.6	63.8	66.0	49.9	64.4	62.4	61.6
	12B	63.9	66.9	66.4	68.0	63.5	63.3	66.4	66.2
TC	Q	41.1	52.0	56.5	57.1	43.5	59.9	61.6	54.2
	M	12.2	16.9	18.6	11.9	12.2	16.4	16.9	14.7

Appendix

Table 4: The two budget knobs. Accuracy (%, 3-repeat mean) when cutting rollout count

k

(

k{=}8

is the full method) and expert budget

B

, the top

B\%

of MI-ranked decisions (

B{=}100\%

is full). Each panel carries its own base , pooled over that panel’s settings because the gate re-measures the unaided agent in every run, and bold marks the best setting per row. In the mean-gain row, — marks the base columns, which have no gain by definition. All 36 results are complete at 3 repeats.

Figure 5: Gain over the re-measured greedy base per training iteration , one panel per benchmark and one line per agent model. Solid lines are 3-iteration moving averages and faded dots are raw 3-repetition means.

InterCode-SQL ( $n{=}150$ )
Method	Qwen3-32B	Ministral-3B	Gemma-4B	Gemma-12B
base	66.8	50.9	47.6	64.6
ACE	66.7	47.3	48.0	64.7
ReasoningBank	65.3	50.7	30.0	67.3
AWM	66.7	56.7	34.0	65.3
AWM-iter	66.7	56.7	38.0	64.0

Appendix

Table 5: Training-time methods with an equal expert , test accuracy (%, 3-repeat mean), referenced from § 5.1 . The shaded rows are ours. base for SynAct is pooled over its complete iterations, and each baseline reports its own paired greedy base. SynAct appears after one training iteration and at its best iteration, with the best iteration in parentheses. Bold marks the best learning method per column and underline the second best.

Benchmark	Acquisition	Qwen3-32B	Ministral-3-3B	Gemma-3-4B	Gemma-3-12B
SQL	base	64.4	49.1	45.8	64.9
	SynAct-Rand (random)	69.3	64.9	50.0	67.3
	SynAct-Ent ( $H(A)$ )	69.3	62.2	51.3	65.6
	SynAct-Succ ( $I(A;\mathrm{succ})$ )	68.2	61.3	49.6	65.3
	SynAct ( $I(A;O)$ )	75.6	65.1	52.9	71.3
TextCraft	base	41.2	13.0	1.7	17.5

Appendix

Table 6: Acquisition-function comparison (accuracy %, 3-repeat mean), referenced from § 5.2 . Everything but the selection rule is held fixed, using the same rollouts, the same authoring call, and the same gate. n/a means the signal acquired too little usable evidence to author a rule at all, a failure mode unique to the success-based score, which requires a task to contain both a winning and a losing rollout. The shaded row is ours, and bold marks the best rule per column.

Benchmark	Model	base	1	2	3	4	5	6	7	8	9	10
InterCode-SQL	Qwen3-32B	66.8	68.0	68.2	69.3	68.4	69.3	68.4	68.2	69.3	69.6	68.4
	Ministral-3-3B	50.9	66.7	66.9	64.7	65.8	68.9	68.9	67.8	67.8	67.8	66.9
	Gemma-3-4B	47.6	50.7	50.9	50.0	50.9	50.9	52.0	49.3	50.7	48.4	49.8
	Gemma-3-12B	64.6	65.6	66.0	66.7	65.1	66.0	65.8	66.7	66.9	66.2	67.6
TextCraft	Qwen3-32B	46.6	58.0	59.8	57.5	61.5	57.5	67.2	64.9	58.6	66.1	67.2
	Ministral-3-3B	13.0	13.2	18.4	19.5	15.5	18.4	14.9	15.5	19.5	16.7	19.5

Appendix

Table 7: SynAct test accuracy (%) at each training iteration (3-repeat mean), grouped by benchmark with one row per agent model, referenced from § 5.3 and Appendix K . The base column pools the unaided measurement over the completed iterations, because the gate re-measures the base each iteration.

	Tokens		Expert
Setting	Agent	Expert	turns
SynAct, 1 iter.	7.6M	0.01M	1
SynAct, 3 iter.	22.9M	0.03M	3
SynAct, 10 iter.	76.2M	0.10M	10
ACE (1 epoch)	0.95M	0.20M	100
ReasoningBank	0.95M	0.15M	50

Appendix

Table 8: Training cost per (model

\times

dataset) run at

n_{\text{train}}{=}50

and

k{=}8

. Expert tokens are the expensive ones, costing

10

–

30\times

the agent’s per-token rate. Token counts are estimates and turn counts are exact.

InterCode-SQL
Model	base	SR(10)	Cascade	Act.-Ent.	Judge	SynAct	+rules
Qwen3-32B	66.0	82.7	70.7	86.0	84.7	85.3	91.3
Ministral-3-3B	47.3	82.0	61.3	88.7	85.9	87.3	88.0
Gemma-3-4B	48.7	62.0	49.3	67.3	66.3	68.0	66.0
Gemma-3-12B	64.7	83.3	65.3	86.0	84.3	85.3	86.7
TextCraft

Appendix

Table 9: Inference-time routing at a matched expert budget (

\theta{=}0.6

R{=}3

), held-out accuracy (%), referenced from § 5.4 . SR(10) is self-reflection with ten retries and no expert. Cascade, action-entropy, and the expert judge are each throttled to SynAct’s expert budget at this operating point. Bold marks the best routing method per row.

Figure 6: Accuracy against retry budget

R

, mean over the four agent models. The no-expert pool with self-reflection against SynAct and SynAct-rules at

\theta{=}0.6

SQL	$R{=}3$	$R{=}10$	TextCraft	$R{=}3$	$R{=}10$
Qwen3-32B	85.3	86.0	Qwen3-32B	81.0	82.8
Ministral-3-3B	87.3	87.3	Ministral-3-3B	29.3	31.0
Gemma-3-4B	68.0	74.0	Gemma-3-4B	12.1	12.1
Gemma-3-12B	85.3	88.7	Gemma-3-12B	50.0	53.4
Bash	$R{=}3$	$R{=}10$	$\tau$ -bench	$R{=}3$	$R{=}10$
Qwen3-32B	67.5	67.5	Qwen3-32B	69.6	70.4

Appendix

Table 10: SynAct at

R{=}10

against

R{=}3

, same expert budget and operating point (

\theta{=}0.6

), accuracy (%).

Model	MI-out	Act.-Ent.	$B$	MI-out	Act.-Ent.
	Full benchmark (acc. %)			Rescues on routed set
Qwen3-32B	69.6	67.8	33	7/33 (21.2)	5/33 (15.2)
Ministral-3-3B	60.9	58.3	44	9/44 (20.5)	6/44 (13.6)
Gemma-3-12B	38.3	33.9	61	8/61 (13.1)	3/61 (4.9)

Appendix

Table 11: Outcome-MI against action entropy as the routing signal (

\tau

-bench retail,

\theta{=}0.6

R{=}3

n{=}115

). Left: full-benchmark accuracy (%). Right: expert rescues among the

B

tasks both signals route, solved/routed (%). Both signals route the same

B

tasks per model, and the rescue columns isolate where the expert plan is anchored. Bold marks the better signal per view.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 590 papers

Oct 2, 2026

Sentry: Learning to Recover from LLM Agent Failures at Test Time

LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.

Changxiu Ji, Amy Lu, Qizheng Zhang +1

Department of Computer Science, Stanford University

LLM Agent Reliability Memory-Augmented Language Model Agents

Ask PDF

Aug 25, 2026

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.

Xiaolong Jin, Dingmin Wang, Vijay Lingam +1

AWS AI Labs · Purdue University

RL for LLM Agents LLM-Guided RL Multi-Turn RL Step-Level Credit Assignment in RL Self-Distillation

Ask PDF

Oct 14, 2026

Beyond Corrected Actions: Training LLM Agents on Textual Critiques

Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains

τ^2

-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.

Thomas Zeng, Zelei Cheng, Mingyang Zhou +3

University of Wisconsin–Madison · AI Foundations, Capital One

LLM Agent Training Language Model Distillation

Ask PDF

Abstract

LLM Agent Training

Explore similar work

Sentry: Learning to Recover from LLM Agent Failures at Test Time

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

Beyond Corrected Actions: Training LLM Agents on Textual Critiques