Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning

Zixuan Wang; Yanrui Miao; Zhengxi Lu; Teng Pan; Yiwen Qiu; Hongxing Li; Peng Qiu; Ruiqing Zhang

cs.AIAug 24, 2026

Agent-G $^2$ : Gaussian Guidance for Agentic Reinforcement Learning

Authors: Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang,

Organizations: Zhejiang University · Baidu Inc. · Shandong University

Abstract

Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G $^2$ , a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G $^2$ on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G $^2$ outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.

Figures & tables

Figure 1: Hint-based RL paradigms across training steps $t_{0},t_{1},t_{2}$ . (a) Schedule-based: shared $d_{t}$ across samples. (b) Probe-based: $O(\log n)$ per-sample probes. (c) Agent-G 2 (ours): $d_{i}\sim\mathcal{N}(\mu_{t},\sigma_{t}^{2})$ per task, estimated online from existing rollouts.

Figure 2: Scalar-depth schedulers misallocate guidance. (a) Shared-depth schedulers keep fewer than half of assignments in-range; (b) per-sample probing lowers the mismatch ratio

\rho

only by spending extra rollouts.

Figure 3: Useful guidance forms a band around

d_{i}^{\star}

. (a) Rollout success rate

p_{i}(d)

vs. depth, with under-/in-range/over-guided regimes shaded; the in-range band spans several depths. (b) Bernoulli-variance informativeness aligned by

\Delta d=d-d_{i}^{\star}

, with a Gaussian fit (

\sigma{=}0.22

R^{2}{=}0.92

Figure 4: Agent-G 2 pipeline. (1) Tasks are clustered offline by difficulty. (2) A global baseline $\mu_{\text{global}}$ and per-cluster statistics $(A_{k},V_{k})$ form a per-task Gaussian $\mathcal{N}(\mu_{i},\sigma_{i}^{2})$ . (3) One prefix length is drawn per task; $R$ rollouts run from the post-prefix state. (4) The policy is updated with prefix SFT plus GRPO loss. (5) The same rollouts refresh $\mu_{\text{global}},A_{k},V_{k}$ for the next batch.

Type	Method	ALFWorld							WebShop
		Short		Medium			Long	All	Score	Succ
		Pick	Look	Clean	Heat	Cool	Pick2	All
Closed-Source Models
Prompting	GPT-4o ( OpenAI et al. 2024 )	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Prompting	Gemini-2.5-Pro ( Comanici et al. 2025 )	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Base: Qwen2.5-1.5B-Instruct

Table 1: Main results on ALFWorld and WebShop (success rate, %). Best and second-best are highlighted.

Figure 5: Beyond imitation. (a) ALFWorld success on Qwen2.5-1.5B: Agent-G 2 (95%) far exceeds Full SFT (56%) and Sampled-Prefix SFT (27%). (b) Per-cluster (Short / Medium / Long) depth densities at

t{=}29

Figure 6: Training dynamics of Agent-G 2 . (a) Success rate vs. gradient steps for Agent-G 2 and the strongest scalar-depth baseline. (b) Evolution of the per-task guidance-depth distribution across training.

Method	Type	Cost/step (s)	Ratio
Agent-G 2	Distribution	$88$	$1.00\times$
Step decay	Schedule	$57$	$0.65\times$
Cosine decay	Schedule	$60$	$0.68\times$
Linear decay	Schedule	$61$	$0.69\times$
Target acc	Schedule	$80$	$0.91\times$
Enumeration	Probe	$425$	$4.83\times$

Table 2: Per-step training cost on Qwen2.5-1.5B / ALFWorld. Median wall-clock seconds per gradient step. Ratios are relative to Agent-G 2 .

Variant	Long	All	$\boldsymbol{\Delta}$
Agent-G 2	94.7	95.3	–
Sampling form
w/o sampling: $d_{i,j}{=}\mu_{i},\sigma_{i}{=}0$	84.6	89.8	$\downarrow\,5.5$
Uniform sampling: $\mathcal{U}[\mu_{i}\!\pm\!\sigma_{i}]$	80.8	88.3	$\downarrow\,7.0$
Adaptive moments
w/o adaptive center: $\mu_{i}{=}\mu_{\mathrm{global}}$	79.2	91.4	$\downarrow\,3.9$

Table 3: Ablation of Agent-G 2 on ALFWorld / Qwen2.5-1.5B-Instruct.

\Delta

is the change in All relative to the full method. Removing

\mathcal{L}_{\mathrm{GRPO}}

is equivalent to Sampled-Prefix SFT.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Step	$n$	Fix-step	Linear	Step-dec	Tar-acc
$10$	$28$	$67.9$	$42.9$	$42.9$	$32.1$
$20$	$28$	$64.3$	$39.3$	$39.3$	$35.7$
$40$	$28$	$75.0$	$57.1$	$57.1$	$50.0$
$80$	$58$	$48.3$	$55.2$	$55.2$	$41.4$
$160$	$45$	$71.1$	$55.6$	$55.6$	$40.0$
Avg	—	$\mathbf{65.3}$	$\mathbf{50.0}$	$\mathbf{50.0}$	$\mathbf{39.8}$

Appendix

Table 4: Mismatch ratio

\rho

of shared-depth schedulers across five training checkpoints on Qwen2.5-1.5B / ALFWorld.

n

is the number of diagnostic tasks at that step.

Hyperparameter	Symbol	Value
Agent-G 2 -specific
Global baseline step	$\Delta$	$0.1$
Cluster EMA rate	$\alpha$	$0.2$
Center scale	$\lambda$	$1.0$
Variance scale	$\gamma$	$1.0$
Variance floor	$\sigma_{\min}$	$0.1$

Appendix

Table 5: Agent-G 2 hyperparameters. Shared across ALFWorld and WebShop unless noted. Hint-based RL baselines reuse the Policy optimization , Rollout , and Training schedule blocks.

Training Stage	$\sigma$	$R^{2}$	$n$
Early (ckpts $0$ – $50$ )	$0.198$	$0.906$	$116$
Mid (ckpts $50$ – $100$ )	$0.206$	$0.888$	$146$
Late (ckpts $100$ – $150$ )	$0.231$	$0.936$	$109$

Appendix

Table 6: Gaussian fit of the aligned informativeness profile across training windows on Qwen2.5-1.5B / ALFWorld. Each window pools

(\Delta d,\,p_{i}(d)(1-p_{i}(d)))

pairs collected within the listed checkpoint range;

n

counts pairs entering the least-squares fit.

Figure 7: Four-phase snapshot of the per-cluster Gaussian schedule on Qwen2.5-1.5B / ALFWorld. Each panel shows the density

\mathcal{N}(\mu_{k},\sigma_{k}^{2})

for the

K{=}3

horizon clusters (Short, Medium, Long) at one training step. Their batch-weighted mixture yields the aggregate in Figure 6 (b).

Imitation
Method	Configuration
Full SFT	LR $1{\times}10^{-5}$ , $\|\mathcal{B}\|{=}32$ , $2$ epochs
RL without hints
GRPO	Table 5 with $d_{i}{=}0$
GiGPO	authors’ default
BEACON	authors’ default

Appendix

Table 7: Baseline configurations. Hint-based RL baselines reuse the training blocks of Table 5 ; only the depth-assignment rule differs.

Figure 8: Per-cluster schedule trajectories on Qwen2.5-1.5B / ALFWorld. Each row shows one horizon cluster (

k_{1}

Short,

k_{2}

Medium,

k_{3}

Long); columns plot

\mu_{t}

and

\sigma_{t}

over training steps.

Clustering signal	$K$	All
None	$1$	$89.1$
Length quantile	$3$	$95.3_{\pm 2.3}$
Length quantile	$5$	$96.5_{\pm 1.9}$
Length quantile	$10$	$86.2$
Random	$3$	$89.4$

Appendix

Table 8: Clustering design ablation on Qwen2.5-1.5B / ALFWorld (success rate %). Length-quantile bins tasks by horizon; random samples a partition uniformly.

\pm

values are

5

-seed standard deviations.

K{=}1

and length-quantile

K{=}3

are reproduced from Table 3 .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,742 papers

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
Aug 7, 2026Haoyu Zheng, Yun Zhu, Qing Wang +1Agentic RL Credit Assignment in RL
Zhejiang University · Tencent · Shanghai AI Laboratory
Learning Agentic Policy from Action Guidance
May 12, 2026Yuxiang Ji, Zengbin Wang, Yong Wang +6Agentic RL Reinforcement Learning
1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group +1
Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning
Sep 11, 2026Taoran Liang, Yang Liu, Shang Luo +10Credit Assignment in RL Step-Level Credit Assignment in RL
Nankai University · Peking University · Supply Chain Tech Team Y, JD.com +2

Aug 7, 2026J/K move · Enter open · S save

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Haoyu Zheng, Yun Zhu, Qing Wang, Wenqiao Zhang

Zhejiang University · Tencent · Shanghai AI Laboratory

Agentic RL Credit Assignment in RL Sparse-Reward RL Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.

Abstract

Reinforcement Learning

Explore similar work

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

Learning Agentic Policy from Action Guidance

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning