Learning to Steer LLM Agents and Multi-Agent Systems

Leyang Shen; Yaxin Gong; Yang Zhang; Chenxiao Fan; Jiakai Wang; Dong Wang; Yang Liu; Fuli Feng

cs.LGOct 15, 2026

Learning to Steer LLM Agents and Multi-Agent Systems

Authors: Leyang Shen, Yaxin Gong, Yang Zhang, Chenxiao Fan, Jiakai Wang, Dong Wang, Yang Liu, Fuli Feng,

Organizations: National University of Singapore · Qwen Business Unit of Alibaba · University of Science and Technology of China

Abstract

Large language model (LLM)-based agents and multi-agent systems increasingly tackle complex tasks through long-horizon tool use and inter-agent collaboration. However, these systems are prone to deviating from the overall goal as execution unfolds, which wastes execution resources and leads to undesired outcomes, a failure termed goal drift. In this work, we propose a steering agent that supervises execution on the fly and steers the system back on track when drift occurs. The steering agent observes the execution context, deciding when to intervene with a low-cost fast path and how to intervene with a high-cost slow path. To unlock its steering intelligence, the ability to assess goal drift and generate effective interventions, we jointly train the two paths with reinforcement learning, combining a pairwise ranking objective with group-relative policy optimization (GRPO). Experiments on information-seeking search tasks show that the steering agent provides effective supervision for both single-agent and multi-agent systems, improving their performance and efficiency. The learned steering intelligence generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency without retraining.

Figures & tables

Figure 1: Scheme comparison of planning, verification, and steering. (a) Execution can deviate from an initial plan. (b) Verification can cause costly retries. (c) Steering provides on-the-fly intervention to keep the system on track.

Figure 2: Goal drift example in single-agent and multi-agent executions. (a) The agent keeps investigating the same topic after obtaining enough information, wasting execution on unproductive actions. (b) The verifier repeatedly rejects answers the searchers have already delivered, sending execution into repeated revision rounds.

Figure 3: Steering agent and reinforcement learning pipeline. (a) The steering agent decides whether to intervene with the fast path and how to intervene with the slow path. (b) The RL pipeline samples states from previous rollouts, samples several steering actions at each selected state, and jointly updates the two paths with a pairwise ranking objective and GRPO.

Method	Multi-Hop QA				Single-Hop QA			Overall		Cost
Method	2Wiki	Hotpot	Bamboogle	MuSiQue	NQ	Trivia	Pop	Avg.	$\Delta$	#Act.	Turns
No Intervention	36.9	46.9	38.4	20.8	46.8	60.6	48.5	42.70	–	–	6.4
Planning	38.1	45.0	43.6	21.9	47.9	61.3	48.4	43.74	+1.04	1.00	8.0
w/ RL	46.5	49.7	45.9	22.9	47.5	61.4	47.5	45.91	+3.21	1.00	8.8
Verification	35.2	43.8	44.6	19.3	44.1	60.8	43.2	41.57	$-1.13$	1.22	7.8
w/ RL	40.8	51.1	39.9	20.9	51.1	61.4	48.9	44.87	+2.17	1.08	6.8

Table 1: Results on in-domain benchmarks.

\Delta

is the gain in Avg. over No Intervention, and Cost reports steering actions (#Act.) and executor turns (Turns) per task. Best results are in bold.

Method	Frames		GAIA		xBench-DS
Method	Avg@4	Pass@4	Avg@4	Pass@4	Avg@4	Pass@4
No Intervention	26.46	44.17	16.75	34.95	49.50	65.00
Steering w/ RL	36.32	59.71	23.79	39.81	49.50	72.00

Table 2: Results on out-of-domain benchmarks with an online search API. We report mean (Avg@4) and best-of-n (Pass@4) accuracy across 4 random seeds. Best results are in bold.

Figure 4: F1 improvement as the intervention frequency varies. By adjusting the fast path threshold without retraining, RL steering maintains the largest total gain (left) and the largest gain per intervention at low frequency (right), prioritizing the states that influence the final outcome most.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: In-domain gains under varied rollout hyperparameters. Each configuration is labelled by

(m,k,G,N)

, the number of selection rounds, states per round, steering actions per state, and continuations per branch.

\Delta

F1 is the paired gain in mean F1 over the seven benchmarks against the no-intervention baseline.

Figure 6: Average latency per call of the fast and slow paths. The two paths share the same prefill cost, and the cost of the slow path comes from generating the reasoning and steering action.

Method	Multi-Hop QA				Single-Hop QA			Overall
Method	2Wiki	Hotpot	Bamboogle	MuSiQue	NQ	Trivia	Pop	Avg.	$\Delta$
No Intervention	44.6	55.7	38.4	17.0	53.7	72.8	47.4	47.09	–
Planning	44.6	53.8	48.0	20.4	55.2	74.7	49.3	49.43	+2.34
w/ RL	48.4	53.5	45.6	18.8	52.0	72.4	47.4	48.30	+1.21
Verification	42.8	52.2	45.6	16.2	50.8	74.2	43.0	46.40	$-0.69$
w/ RL	43.1	55.1	42.4	19.7	56.3	71.7	47.7	48.00	+0.91

Appendix

Table 5: LLM-as-a-Judge results on in-domain benchmarks. The methods and predictions are the same as in Table 1 , and

\Delta

is the gain in Avg. over No Intervention. Best results are in bold.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 711 papers

From Uncertainty to Action: Learning to Steer LLM Agents
Oct 6, 2026Hanwen Li, Jinhao Duan, Guanhua Zhu +4Language Model Steering LLM Agents
New Jersey Institute of Technology · University of North Carolina at Chapel Hill · University of California, Irvine +1
Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systems
Feb 24, 2026Mohammad Parsa Hosseini, Ankit Shah, Saiyra Qureshi +3Multi-Agent LLM Systems LLM Agent Orchestration
Accenture.
Multi $^2$ : Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments
Jun 2, 2026Sangeun Park, Minhae KwonMulti-Agent LLM Systems Hierarchical RL
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea.

Oct 6, 2026J/K move · Enter open · S save

From Uncertainty to Action: Learning to Steer LLM Agents

Hanwen Li, Jinhao Duan, Guanhua Zhu, Junchi Lu+3

New Jersey Institute of Technology · University of North Carolina at Chapel Hill · University of California, Irvine+1

Language Model Steering LLM Agents

Open paper AskPDF arXiv

Steering an LLM agent means deciding whether to correct it, at which step, and with which mechanism. Uncertainty is often used to decide when to correct an agent, but whether it can guide these decisions remains unclear. We steer agent trajectories separately at every non-terminal step with each of four mechanisms and run each continuation to completion. The resulting stepwise outcome table (SOT) holds about 82,000 counterfactual continuations of 1,864 trajectories from three benchmarks and two agents. It shows that uncertainty can identify failing trajectories, but that no single signal reliably locates the step at which steering helps. We therefore propose VoS (Value of Steering), a trajectory-level monitor, offline or online, that learns from SOT the value of steering at each step and decides where to steer by it. A harm-budgeted trigger decides whether to steer, limiting the fraction of successful trajectories that VoS disturbs. VoS improves on unmodified execution in all 12 settings of benchmark, agent, and offline or online use, by 7.8 points on average, and outperforms the strongest of five existing uncertainty-triggered methods in 11, by 2.9 points on average. Ablations show that training on measured outcomes and a tight harm budget are both essential.

Abstract

LLM Agents

Explore similar work

From Uncertainty to Action: Learning to Steer LLM Agents

Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systems

Multi2^22: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

Multi $^2$ : Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments