Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics

O. Ozan Koyluoglu

cs.AIOct 9, 2026

Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics

Authors: O. Ozan Koyluoglu

Organizations: Samaya AI

Abstract

Language-model agents are used for specialized work such as financial research, healthcare and law. Such agents are often capable enough for a task but still miss its details. In finance, for example, these details include which filing reports a figure or how a company discloses a metric. Retraining the agent to learn these details is expensive, and often impossible when the agent is a closed or shared system. Writing the missing know-how into the agent's prompt makes the prompt long and identical for every input, although different inputs need different guidance. We propose to learn to generate this guidance instead. A small advisor model reads each input and writes structured advice, which is appended to the prompt of a frozen executor. The advisor is trained with hindsight, following work on distillation with privileged information. On a training pool with graded answers, we analyse the executor's failures, run it under several strategies to see which one worked on each input, and let a teacher that sees the grades and the traces write the advice that would have helped. The advisor then learns, by supervised fine-tuning, to write that advice from the input alone. Because the student is a separate small model rather than the executor, it is cheap to train and adds little latency. We call this approach hindsight structured advice distillation (HSAD). On FrontierFinance, a public benchmark of expert-written financial research queries, an 8B advisor raises the qualification rate of Samaya's proprietary research agent, which already leads the benchmark's leaderboard, from 52.9% to 56.6%. This is above the agent's own high-effort configuration, at 37% lower cost per query. We study the generalization of the proposed approach, and in particular show that HSAD also improves results on WebShop, BIRD text-to-SQL and ExpertLongBench.

Figures & tables

Domain	Field	computed by	computed from	how	produced by
		Training label			At test time
WebShop	arm	rule-based	arm scores	highest-scoring arm; ties to default	advisor
	effort , expected_tools	rule-based	winner’s trace	winner’s tool calls	advisor
	required_options , directive_picks	rule-based	input	options stated in the instruction	advisor
FrontierFinance	arm	rule-based	arm scores	highest-scoring arm; cascade tie-break	advisor
	target_length , web_use , sources , tools	LLM reasoning	gold + all runs	reads all arms’ answers, grades	advisor

Table 1: Source of each advice field’s training label: how the teacher computes it (rule-based or by LLM reasoning) and what it is computed from. “Arm scores” are the grader’s scores of every arm’s output against gold; “winner’s trace” is the run of the winning arm. The last column shows what produces each field at test time.

Setting	Frozen executor	Metric	Train	Test	Arms	Directives
FrontierFinance	Samaya production agent	rubric qualification rate	1,819	220	4	103
WebShop	Qwen3-4B agent	score (0–100), success rate	6,210	500	6	6
BIRD	Qwen3-4B	execution accuracy	9,421	1,074	9	63
ExpertLongBench	Qwen3-4B	checklist coverage	450	100	8	57

Table 2: The four settings. “Train” is the number of advisor training rows. Arms include default .

System (harness)	Qualification rate	Cost / query	Latency / query
Samaya System + advisor	56.6%	$1.14	257 s
Samaya System, high effort	56.0%	$1.81	278 s
Claude Fable 5.1 (Finance Agent v2)	55.9%	$6.89	268 s
Samaya System (no advice)	52.9%	$0.93	214 s
Claude Fable 5 (Finance Agent v2)	49.2%	$4.06	165 s
GPT 5.6 Sol (Finance Agent v2)	46.8%	$3.03	171 s

Table 3: FrontierFinance public leaderboard (220 queries, all rubric items) with the advised agent added. All rows except the advised agent are taken from the public FrontierFinance leaderboard [ 52 ] . The two Samaya rows from the leaderboard use the same proprietary harness we use, the other rows use the open-source Finance Agent v2 harness.

Teacher advice	essential	all	Contribution (essential)
all fields	69.3%	59.6%	–
without directive_picks	62.6%	54.3%	6.7pp
without sources	64.9%	55.5%	4.4pp
without tools plan	66.3%	56.2%	3.0pp
without web_use	66.6%	57.0%	2.7pp
without target_length	67.5%	57.5%	1.8pp

Table 4: Leave-one-field-out ablation of the teacher’s advice on the FrontierFinance validation set. The contribution of a field is the drop in essential-criteria coverage when it is removed.

Setting	Metric	No advice	With advice	Lift
WebShop	score / success rate	54.62 / 3.8%	63.35 / 23.2%	+8.73 / +19.4
BIRD	execution accuracy	40.04%	43.11%	+3.07
ExpertLongBench	checklist coverage	49.7%	54.5%	+4.9

Table 5: Held-out lift of the trained advisor with a frozen Qwen3-4B executor. Each row compares the same executor on the same inputs with and without advice. Success-rate lifts are in percentage points.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Arms
WebShop	default , concise_search_and_fast_commit , strict_option_click_then_buy , exact_variant_text_verification , category_demographic_and_price_verify , bounded_variant_fallback_pivot
FrontierFinance	default , structured_priority , tables_extraction , restricted_tools
BIRD	default , evidence_literal_and_formula_execution , schema_grounding_and_minimal_joins , conditional_aggregation_and_ratios , superlative_extremums_and_ties , cte_multi_stage_decomposition , strict_projection_and_output_alignment , sqlite_dialect_and_data_sanitization , multi_intent_union_and_cardinality
ExpertLongBench	default , schema_first_structured_output , entity_identity_first_then_annotate , security_threat_frame_first , pedagogical_evaluator_role_lock , mechanistic_rationale_extraction , clinical_note_completeness_and_fidelity , exhaustive_field_coverage_guard

Appendix

Table 6: Arms of each setting. Each arm is appended to the domain’s base prompt, and default adds nothing.

Training variants of the advisor
	Public test (220)		Validation (100)
Condition	essential	all	essential	all
No advice	53.9%	45.4%	63.0%	53.9%
Best fixed arm ( tables_extraction )	57.9%	51.7%	69.1%	60.2%
Trained advisor	59.9%	52.6%	69.2%	61.9%
more epochs (9)	59.3%	52.2%	65.9%	58.4%

Appendix

Table 7: FrontierFinance in the deployment configuration. The best fixed arm applies the same arm to every question (without other fields in the advice).

Task	$n$	Lift	Most frequent predicted arm	Example directive picked
T11 security review	18	+20.4	security_threat_frame_first (12)	8.1 detect_and_label_prompt_injection
T04 pedagogical evaluation	19	+9.0	pedagogical_evaluator_role_lock (19)	5.3 separate_mistake_identification…
T03 materials synthesis	9	0.0	security_threat_frame_first (7)	3.8 capture_precursor_rationale…
T08 protein descriptions	18	0.0	clinical_note_completeness… (16)	3.6 populate_biological_annotation…
T06 clinical notes	18	$-1.0$	security_threat_frame_first (18)	3.5 transcribe_complete_examination…
T07 molecule descriptions	18	$-1.9$	pedagogical_evaluator_role_lock (13)	3.7 report_all_compound_classification…

Appendix

Table 8: ExpertLongBench lift by task on the 100 test documents, in checklist-coverage points, with the arm the advisor chose most often for that task (count in parentheses) and a representative directive it picked.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 711 papers

Sep 29, 2026

AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

A small trainable advisor can steer a frozen language-model executor using natural-language advice. In addition to learning from task rewards, the advisor can use feedback from completed interactions to improve its advice. However, a plausible correction need not change execution, yet learning from such corrections can still affect the advisor's future decisions in other contexts. In a shared-parameter model, we prove that such corrections can limit learning if their targets favor useful advice less strongly than those of other corrections. Keeping them less often than the rest improves the model's eventual performance compared to learning from every correction. Motivated by this, our method, Advisor Self-Distillation (AdviSD), pairs outcome-based reinforcement learning with self-distillation from a feedback-conditioned copy of the advisor selectively. Reflection proposes corrections, and the advisor scores the same recorded executor response with and without its issued advice, using the magnitude of the difference to select decisions for supervision. This approach does not require executor likelihoods or additional executor rollouts. Experiments with Qwen3-8B advisors for Gemini and Claude show that AdviSD outperforms advisor-GRPO by 4.2-6.4 percentage points on BFCL-v3 and by 3.9-5.1 score points on EnvScaler. The trained advisors generalize to out-of-domain tasks and transfer across different executor versions and model families. AdviSD also beats matched-count random selection, supporting the value of its selection rule.

Rishabh Agrawal, Hejie Cui, Shasha Li +2

Google · University of Southern California

Language Model Steering LLM Agent Self-Improvement Self-Distillation RL for LLM Agents

Ask PDF

Oct 7, 2026

Training Advisors for LLM Agents from Task Outcomes

Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including

τ^3

and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.

Sergei Polezhaev, Barys Liskavets, Ori Press +1

Nebius AI

LLM Agent Training RL for LLM Agents

Ask PDF

Oct 24, 2026

ReGE: Stronger Agents through Experience Synthesis

Repeated attempts at interactive tasks uncover facts, useful operations, and failures that can inform subsequent execution. We introduce ReGE, a framework for turning this recorded experience into guidance for a fresh attempt. Its main implementation, ReGE-Syn, uses a single coach call to jointly synthesize bounded execution records and available artifacts into self-contained guidance, connecting task-specific findings with proposed actions and relevant pitfalls. A student starts from the task's initial environment state and adapts this guidance to current observations. Source generation, synthesis, and student execution use the same base model, without parameter updates or a stronger teacher. Across four benchmarks, ReGE improves accuracy by an average of 9.9 percentage points for GPT and 8.8 percentage points for Claude compared with unguided execution. On matched BrowseComp-Plus subsets, ReGE-Syn reduces guidance-construction tokens by 98.2% on GPT and 95.5% on Claude relative to adapted AggAgent, with observed success. Combined construction and execution tokens decrease by 15.7% and 35.4%, respectively, counting input plus output and excluding source generation. BrowseComp-Plus ablations further associate access to process information with fewer student steps and show that joint synthesis produces substantially shorter guidance than Direct history or Independent summaries, with observed success. These results demonstrate how agents can use their own recorded experience to improve subsequent task completion, and suggest a low-cost way to consolidate the execution records of parallel agent systems such as agent swarms.

Jiarong Liang, Yubo Wang, Zijie Liu +3

University of Waterloo · University of North Carolina at Chapel Hill · University of British Columbia

LLM Agent Self-Improvement LLM Agent Trajectory Synthesis LLM Agent Efficiency

Ask PDF

Abstract

LLM Agents

Explore similar work

AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

Training Advisors for LLM Agents from Task Outcomes

ReGE: Stronger Agents through Experience Synthesis