CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu; Zhengcheng Wang; Jiahong Liu; Zexuan Qiu; Minda Hu; Shaohang Wei; Jiafei Lyu; Junyan Lyu

cs.LGOct 21, 2026

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Authors: Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu,

Organizations: The Chinese University of Hong Kong · Tencent · Peking University

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.

Figures & tables

Figure 1: From gradient conflict to improved agent training. Left, observations (a) and (c): cosine quantiles of conflicting WebShop/Qwen3-1.7B gradient records ( Qwen Team, 2025 ) , before and after analytic CORD projection. Middle, observation (b): first-order RL-surrogate progress relative to RL alone on the same log, evaluated at

\lambda=0.005

with RL protection. Right: preliminary WebShop/3B training success with

k_{3}

\lambda=0.001

, and conflict-only distillation protection. The right panel shows iterations 75–148 after 21-iteration smoothing. Curves average two training seeds and bands show their range. Appendix C.1 gives measurement and smoothing details.

Figure 2: CORD combines two learning signals through gradient projection. Left: GRPO uses environment rewards, while OPD uses a teacher conditioned on privileged skill context, with teacher gradients disabled. Middle: accumulated gradients $\bm{u}=\nabla F_{1}$ and $\bm{v}=\lambda\nabla F_{2}$ give the DirectSum $\bm{u}+\bm{v}$ . For the conflicting pair shown, CORD protects $\bm{u}$ and removes the opposing component of $\bm{v}$ , giving $\bm{u}+\tilde{\bm{v}}$ . Either objective can be protected. Right: illustrative paths in an analytic two-loss construction with GRPO protection. Height and shading encode its GRPO loss. Arrows denote updates of step size $0.7$ , stopping within $0.10$ of either local minimum.

	ALFWorld							WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	Success Rate
Vanilla †	$44.4$	$11.1$	$6.2$	$15.4$	$28.6$	$12.5$	$21.9$	$6.7$	$0.8$
Skill-Prompt ∗†	$51.7$	$66.7$	$48.4$	$0.0$	$4.3$	$10.0$	$28.9$	$0.2$	$0.8$
OPSD †	$48.8$	$41.7$	$16.7$	$0.0$	$15.8$	$16.7$	$28.1$	$11.3$	$3.1$
GRPO	$93.5{\scriptstyle\pm 3.1}$	$72.9{\scriptstyle\pm 12.3}$	$67.0{\scriptstyle\pm 3.0}$	$62.5{\scriptstyle\pm 5.9}$	$59.5{\scriptstyle\pm 2.4}$	$66.3{\scriptstyle\pm 8.3}$	$71.3{\scriptstyle\pm 1.9}$	$71.1{\scriptstyle\pm 1.1}$	$59.2{\scriptstyle\pm 1.5}$
RLSD	$95.4{\scriptstyle\pm 1.6}$	$68.8{\scriptstyle\pm 3.6}$	$83.0{\scriptstyle\pm 3.9}$	$\mathbf{87.5}{\scriptstyle\pm 7.8}$	$56.0{\scriptstyle\pm 3.9}$	$53.8{\scriptstyle\pm 6.7}$	$74.4{\scriptstyle\pm 2.2}$	$69.8{\scriptstyle\pm 0.7}$	$54.4{\scriptstyle\pm 1.0}$

Table 1: Final results on Qwen2.5-3B-Instruct. ALFWorld success and WebShop score/success are on a 0–100 scale.

\dagger

: Lu et al. (2026) .

*

: skills at inference. Our step-150 results use one training seed and four evaluation passes.

\pm

denotes decode standard deviation. CORD protects RL with conflict-only projection.

\Delta=\mathrm{CORD}-\mathrm{Base}

uses displayed means: success differences are percentage points and score differences are score points, with red/green indicating increases/decreases. Deltas have no assigned standard deviation and are excluded from ranking. Bold/underline mark the largest/second-largest distinct method means. Ties share the mark.

Projection rule	Gate	$\lambda=0.001$	$\lambda=0.005$
No projection	—	$55.75$ $\,\pm 0.91$	$56.75$ $\,\pm 0.89$
Protect RL	Conflict-only	$59.35$ $\,\pm 0.79$	$61.85$ $\,\pm 1.59$
Protect RL	Always	$62.65$ $\,\pm 0.79$	$61.25$ $\,\pm 0.92$
Protect distillation	Conflict-only	$58.35$ $\,\pm 0.74$	$61.90$ $\,\pm 1.08$
Protect distillation	Always	$38.90$ $\,\pm 0.95$	$62.10$ $\,\pm 1.54$
PCGrad ( Yu et al., 2020 )	Conflict-only	$63.10$ $\,\pm 0.91$	$58.85$ $\,\pm 0.95$

Table 2: Protection direction and gating on WebShop. Qwen2.5-3B-Instruct with

k_{3}

, reporting success (%). Values are means and population standard deviations across four evaluation passes at temperature

0.4

after 150 updates, with one training seed. Protect RL projects the distillation gradient. Protect distillation projects the RL gradient. PCGrad projects both on conflict. Bold identifies the largest observed mean at each coefficient.

	ALFWorld							WebShop
Objective	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	Success
$k_{1}$	$\mathbf{95.71}$ ${\scriptstyle\pm 4.74}$	$71.15$ ${\scriptstyle\pm 6.38}$	$71.30$ ${\scriptstyle\pm 4.81}$	$56.25$ ${\scriptstyle\pm 4.42}$	$67.00$ ${\scriptstyle\pm 9.54}$	$66.67$ ${\scriptstyle\pm 5.10}$	$74.11$ ${\scriptstyle\pm 3.00}$	$73.68$ ${\scriptstyle\pm 0.79}$	$\mathbf{59.95}$ ${\scriptstyle\pm 0.77}$
$\rho_{\mathrm{d}}$	$92.14$ ${\scriptstyle\pm 3.11}$	$65.38$ ${\scriptstyle\pm 3.85}$	$85.19$ ${\scriptstyle\pm 2.62}$	$78.13$ ${\scriptstyle\pm 3.13}$	$70.00$ ${\scriptstyle\pm 2.00}$	$\mathbf{70.83}$ ${\scriptstyle\pm 2.95}$	$79.11$ ${\scriptstyle\pm 1.28}$	$69.13$ ${\scriptstyle\pm 0.73}$	$58.95$ ${\scriptstyle\pm 0.38}$
$k_{3}$	$93.57$ ${\scriptstyle\pm 1.24}$	$\mathbf{80.77}$ ${\scriptstyle\pm 8.60}$	$\mathbf{91.67}$ ${\scriptstyle\pm 3.07}$	$\mathbf{79.69}$ ${\scriptstyle\pm 6.81}$	$\mathbf{85.00}$ ${\scriptstyle\pm 1.73}$	$63.54$ ${\scriptstyle\pm 1.80}$	$\mathbf{83.75}$ ${\scriptstyle\pm 1.06}$	$\mathbf{74.66}$ ${\scriptstyle\pm 0.44}$	$55.55$ ${\scriptstyle\pm 0.79}$

Table 3: Token objectives at

\lambda=0.005

. ALFWorld success and WebShop score/success are on a 0–100 scale. CORD protects RL with conflict-only projection. Values are mean

\pm

standard deviation across four decoding passes of step-150 Qwen2.5-3B-Instruct checkpoints at temperature

0.4

, with one training seed. Bold marks the largest mean per column.

Figure 3: Effect of

\lambda

on WebShop success. Error bars show standard deviation across decoding passes.

Aggregation	$B$	No projection	Protect RL	Protect distillation
Microbatch	64	53.98	57.55	57.42
Global token	32	52.52	57.46	49.22
Global token	64	59.11	57.33	61.03
Global token	128	61.81	58.10	55.86
Trajectory	64	60.99	61.77	57.42

Table 4: Effect of gradient aggregation and mini-batch size. ALFWorld task-macro success (%) after five continuation rounds from the same Qwen2.5-3B step-90 checkpoint and AdamW state.

B

counts response rows per optimizer mini-batch. Appendix C.4 specifies the single-seed evaluation protocol and aggregation weights. Bold marks row maxima.

Figure 4: Option completion and repetition in WebShop. Qwen2.5-3B-Instruct,

k_{3}

\lambda=0.005

, step 150, conflict-only projection, and greedy decoding. Boxes show selected options and the purchase action, with original step indices. Green highlights selection of the requested color. Dashed amber boxes mark repeated selections. Scores are on

[0,1]

. Success requires

1.00

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Training source	Evaluation source	Turns	Metric
ALFWorld	$3{,}553$ training games	$140$ in-distribution games	$50$	Success rate
WebShop	Goal pool from a $1{,}000$ -product catalogue	$500$ worker–goal assignments	$15$	Score; success

Appendix

Table 5: Task and environment configuration. A turn denotes one generated response followed by an environment interaction. Evaluation sizes describe the source task pools or episode assignments. Section B.3 specifies the fixed evaluation protocol. The WebShop catalogue size counts products, rather than training instructions.

Base recipe	$\lambda$	Confidence weighting $w_{t}$	OPD token term $\psi_{t}$
Skill-SD	$10^{-3}$	$1$ (off)	Importance-weighted $k_{3}$
GRPO+OPSD	$5\!\times\!10^{-3}$	$1$ (off)	$k_{1}$ , without $\rho_{\mathrm{on},t}$
SDAR	$10^{-2}$	$\operatorname{sg}[\sigma(-5\ell_{t})]$	$k_{1}$ , without $\rho_{\mathrm{on},t}$

Appendix

Table 6: Base recipes in the shared implementation. The OPD column specifies the token term, including importance weighting. Every Base configuration disables projection.

Skill context	Guidance
ALFWorld: cleaning	Take the object, clean it at a sink, move to the target location, and place it there.
WebShop: apparel	Select the required size and color, then check the price of the chosen variant before purchasing.
WebShop: search recovery	After two or three unproductive variant changes, return to search instead of continuing the same loop.

Appendix

Table 7: Representative content from the fixed skill bank, paraphrased for brevity. These entries guide the teacher without entering the student’s rollout or evaluation input.

Setting	ALFWorld	WebShop
Training iterations	$150$	$150$
Task groups per rollout	$16$	$16$
Trajectories per group	$8$	$8$
Maximum prompt tokens	$2{,}048$	$4{,}096$
Maximum response tokens per turn	$512$	$512$
History retained (previous turns)	$2$	$2$

Appendix

Table 8: Common training settings for the three base recipes and their CORD variants. Task groups and trajectories are counted before trajectories are expanded into step-level training records. Learning rates are constant after the indicated warm-up.

	ALFWorld							WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg.	Score	Success Rate
GRPO	$95.4{\scriptstyle\pm 3.1}$	$64.6{\scriptstyle\pm 3.6}$	$\mathbf{83.0}{\scriptstyle\pm 5.3}$	$71.4{\scriptstyle\pm 5.1}$	$45.2{\scriptstyle\pm 5.3}$	$65.4{\scriptstyle\pm 2.7}$	$72.9{\scriptstyle\pm 1.2}$	$80.1{\scriptstyle\pm 0.8}$	$69.3{\scriptstyle\pm 1.1}$
RLSD	$95.4{\scriptstyle\pm 3.1}$	$72.9{\scriptstyle\pm 9.1}$	$67.0{\scriptstyle\pm 3.9}$	$42.9{\scriptstyle\pm 5.1}$	$32.1{\scriptstyle\pm 6.2}$	$69.2{\scriptstyle\pm 3.8}$	$65.6{\scriptstyle\pm 1.8}$	$\mathbf{84.4}{\scriptstyle\pm 0.3}$	$\underline{71.0}{\scriptstyle\pm 0.6}$
Skill-SD	$\underline{96.3}{\scriptstyle\pm 2.6}$	$68.8{\scriptstyle\pm 9.1}$	$\underline{80.4}{\scriptstyle\pm 4.0}$	$\mathbf{75.0}{\scriptstyle\pm 6.2}$	$\underline{48.8}{\scriptstyle\pm 3.9}$	$44.2{\scriptstyle\pm 8.4}$	$69.5{\scriptstyle\pm 3.3}$	$\underline{84.2}{\scriptstyle\pm 0.5}$	$\mathbf{71.6}{\scriptstyle\pm 0.7}$
+ CORD	$\mathbf{99.1}{\scriptstyle\pm 1.6}$	$\underline{83.3}{\scriptstyle\pm 5.9}$	$76.8{\scriptstyle\pm 4.0}$	$\underline{73.2}{\scriptstyle\pm 5.9}$	$\mathbf{51.2}{\scriptstyle\pm 3.9}$	$74.0{\scriptstyle\pm 3.2}$	$\mathbf{77.0}{\scriptstyle\pm 1.3}$	$81.3{\scriptstyle\pm 1.3}$	$69.8{\scriptstyle\pm 1.5}$
$\Delta$	$+2.8$	$+14.5$	$-3.6$	$-1.8$	$+2.4$	$+29.8$	$+7.5$	$-2.9$	$-1.8$

Appendix

Table 9: Final results on Qwen2.5-7B-Instruct. ALFWorld success and WebShop score/success are on a 0–100 scale. Results use one training seed and four evaluation passes at temperature

0.4

after update 150. ALFWorld uses 128 fixed assignments and WebShop uses 500 fixed tasks.

\pm

denotes population standard deviation across the four decoding passes. CORD protects RL with conflict-only projection.

\Delta=\mathrm{CORD}-\mathrm{Base}

uses displayed means, in percentage points for success and score points for score. Red/green indicate increases/decreases. Bold/underline mark the largest/second-largest method means.

Figure 5: Effect of the mixing coefficient on WebShop score. The checkpoints and evaluation protocol are the same as in Figure 3 . Error bars show standard deviation across decoding passes.

Figure 6: Task-level projection intervention under local coefficient reweighting. Each line connects the two protection directions for one source task under always projection. Columns reuse the same gradients at different coefficients. The horizontal axis reports the relative change in the combined update. Its labels

U_{0}

and

U_{t}

denote

\bm{d}_{\mathrm{sum}}

and

\bm{d}_{\mathrm{CORD}}

, respectively.

Figure 7: ALFWorld continuation results with task-bootstrap uncertainty. Panel (a) varies mini-batch size under global-token averaging (

L_{2}

), while panel (b) varies aggregation at

B=64

. Error bars show 95% intervals over tasks under the protocol above.

Figure 8: Training success, interaction length, and reference KL. Top: SDAR on WebShop with Qwen2.5-3B (

\lambda=0.01

). Bottom: Skill-SD on ALFWorld with Qwen2.5-7B (

\lambda=0.001

). Columns show rollout success, mean environment turns per episode, and the logged sample-based KL estimate to the frozen reference policy. Faint curves show raw observations, while darker curves show 11-iteration centered moving averages with shorter windows at the boundaries. Each method has one training seed, shown over the common range of iterations 1–148.

Figure 9: Token entropy and response length on ALFWorld. Skill-SD with Qwen2.5-3B at

\lambda=0.001

. Left: token-mean policy entropy on the rollout responses, measured before the actor update. Right: mean response-token count, rather than tokens per complete episode. Each method uses one training seed. The raw overlays, 11-iteration smoothing, and common range of iterations 1–148 follow Figure 8 .

Recorded workload	Shared stages (s)	Base actor (s, est.)	CORD actor (s, est.)	Iteration ratio (est.)
Projection-off run	150.2	54.8	97.2	$1.21\times$
CORD run	147.7	50.0	88.8	$1.20\times$

Appendix

Table 10: Component-based estimates for two WebShop workload profiles. Each row holds the recorded workload fixed for both methods. Shared-stage times are observed means, whereas actor times and iteration ratios are estimates for the ordinary Base and the shared-forward CORD implementation.

Metric	Skill-SD	GRPO+OPSD	SDAR
ALFWorld
Pick	$4.18\!\times\!10^{-1}$	$3.32\!\times\!10^{-1}$	$1.25\!\times\!10^{-1}$
Look	$6.86\!\times\!10^{-2}$	$1.35\!\times\!10^{-1}$	$9.62\!\times\!10^{-3}$
Clean	$5.85\!\times\!10^{-1}$	$5.16\!\times\!10^{-3}$	$8.39\!\times\!10^{-3}$
Heat	$7.67\!\times\!10^{-1}$	$1.79\!\times\!10^{-2}$	$8.01\!\times\!10^{-2}$
Cool	$\mathbf{1.88\!\times\!10^{-4}}^{\dagger}$	$9.15\!\times\!10^{-3}$	$4.23\!\times\!10^{-2}$

Appendix

Table 11: Significance tests for Table 1 . Raw two-sided Welch

p

-values are reported to three significant digits. Bold

\dagger

entries have Holm-adjusted

p<0.05

Figure 10: Additional WebShop trajectories. Trajectories retain the original step indices and use context lines to summarize omitted actions. Green marks requested attributes and amber marks repeated or incomplete actions. Outcomes report the final score and total number of steps, with score

1.00

required for success.

Figure 11: Additional ALFWorld trajectories. (a) Achieving the requested object state. (b) Completing two deliveries. Action labels are abbreviated, with original step indices and context for omitted intervals. “Clock” denotes an alarm clock and “place” denotes the environment’s move action. Green marks goal-relevant manipulation, and amber marks incorrect or ineffective actions.

Figure 12: Token probability changes along a complete trajectory. Legend keys mark row positions, while outlines identify tabulated tokens, not update quality. CORD protects OPD. GRPO protection matches DirectSum at measured precision and is omitted. Dots denote spaces, and tag-boundary pieces are retained. The recorded outcome predates these updates.

Gradient Interference

Gradient interference studies how competing learning objectives produce updates that disrupt one another in shared model parameters.

See all 63 papers

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
Aug 3, 2026Chunji Lv, Yangguang Wei, Junlin Liu +6Language Model Distillation On-Policy Distillation
Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences +1
When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation
Sep 28, 2026Xinke Jiang, Tao Feng, Zhibang Yang +4Gradient Interference Reinforcement Learning with Verifiable Rewards
National Engineering Research Center of Software Engineering, Peking University, Beijing, China · School of Computer Science, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +1
Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
Aug 14, 2026Qizhen Lan, Xi Xiao, Xiangchen Guan +4On-Policy Self-Distillation Agentic RL
The University of Texas Health Science Center at Houston · University of Alabama at Birmingham · Amazon +3

Aug 3, 2026J/K move · Enter open · S save

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Chunji Lv, Yangguang Wei, Junlin Liu, Yang Gao+5

Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences+1

Language Model Distillation On-Policy Distillation Sparse-Reward RL RL for LLM Agents

Open paper AskPDF arXiv

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

Abstract

Gradient Interference

Explore similar work

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation

Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL