ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

Yongqi Tong; Tan Li Hui Faith; Choy Zhen Wen Marcus; Zhou Jin; Kewei Fu; Jiang-Ming Yang; Jianshe Li; Xin Zhang

cs.AIAug 13, 2026

ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

Authors: Yongqi Tong, Tan Li Hui Faith, Choy Zhen Wen Marcus, Zhou Jin, Kewei Fu, Jiang-Ming Yang, Jianshe Li, Xin Zhang

Organizations: Ant International

Abstract

Open-ended real-world interaction admits multiple valid behaviors: an agent may answer directly, ask for clarification, provide progress updates, or confirm before acting. This flexibility breaks a core assumption behind group-based RL: rollouts compared within a group are no longer guaranteed to be behaviorally comparable. As a result, reward-model preferences over interaction style can distort relative advantages and steer optimization toward reward-preferred behaviors rather than context-appropriate ones. We formalize this as a \textit{reward fairness problem} and propose \textbf{ARC} (Advantage Regularization via Conditioning), a training recipe that restores fairer relative comparison through strategy-conditioned rollout grouping, together with hybrid rewards and entropy regularization. We study ARC in our proposed \inter, a novel paradigm for responsive, steerable, and execution-aware user-agent interaction that decouples user-visible communication from latent reasoning and tool use. \inter\ also provides the annotation and distillation pipeline for constructing \inter-86K, our strategy-annotated training corpus for supervised and RL training. Empirically, ARC substantially strengthens the core $τ/τ^2$ tool-use benchmarks, while \inter\ reduces time-to-first-token from 4.91s to 1.27s relative to a think-style baseline. Together, these results suggest that a central bottleneck in open-ended interactive learning is not only how agents are rewarded, but whether their behaviors are compared fairly in the first place. The ARC implementation and \inter-86K training data will be released.

Figures & tables

Figure 1: Overview of ARC. ARC modifies group-based RL by pairing each example with a strategy instruction during training, so rollouts are compared only within strategy-conditioned groups, yielding cleaner relative advantages by eliminating cross-strategy contamination. At inference, the instruction is removed and the policy selects strategies autonomously, making ARC a general recipe for fairer advantage estimation in open-ended interaction.

Figure 2: Standard think-then-answer (left) completes reasoning and tool execution before answering, resulting in high TTFT and no steering, whereas INTER 3 (right) interleaves reasoning, answering, and tool use, allowing real-time steering and faster TTFT.

Figure 3: Latency-capability trade-off for interactive agent architectures. Channel-separated interaction shifts the operating point into a substantially lower-latency regime, while ARC improves capability within that regime rather than causing the horizontal TTFT shift itself. Non-INTER 3 baselines are schematic architectural placements rather than exact benchmark claims.

Category	Strategy	Description
Progress Update	Tool Execute	Execute single or sequential tool calls; return the final answer upon completion
	Parallel Tools	Execute independent tool calls simultaneously; synthesize combined results
	Multi-Step Update	Stream incremental progress updates to the user between sequential sub-tasks
	Silent Execution	Execute tool calls without emitting user-visible output when intermediate steps are irrelevant
	Error Recovery	Handle tool failures gracefully and recover with corrective actions
Clarify First	Clarify First	Ask the user for missing or ambiguous information before executing any tool call

Table 1: INTER 3 strategy taxonomy used for ARC conditioning, analysis, and data construction.

Figure 4: Left: INTER 3 strategy distribution in the RL training dataset. Progress Update dominates at 75.3%, reflecting the prevalence of multi-step tool-use scenarios that benefit from incremental user updates. Right: data domain composition of the full training set (SFT + RL). Tool Use accounts for 72.8% of examples, with Multi-hop QA (19.8%) and Logical Reasoning (7.1%) providing complementary reasoning diversity.

Token Statistics
Statistic	Value
Total examples	86,796
Average input tokens	702.1
Average output tokens	264.5
Average total tokens	966.6
Tool Statistics

Table 2: Inter-86K summary statistics. Token counts use whitespace tokenization.

Figure 5: Training reward curves for GRPO and ARC.

Baselines
Method	Avg.	TTFT $\downarrow$	Tool Calling					Reasoning			Instruction Following	Alignment
Method	Avg.	TTFT $\downarrow$	$\tau$ -airline	$\tau$ -retail	$\tau^{2}$ -airline	$\tau^{2}$ -retail	$\tau^{2}$ -telecom	AIME 2026	GPQA-D	HMMT 2025	IFBench	ArenaHard
Qwen3-8B-noThink	22.85	0.05s	12.00	29.86	14.61	36.55	17.80	17.08	45.71	12.92	24.83	17.14
Qwen3-8B-Think	32.82	4.91s	28.00	36.81	29.75	38.71	23.46	47.92	53.54	20.83	19.43	29.83
RL Backbones
PPO	27.49	0.45s	35.33	41.45	33.33	38.89	19.01	30.83	38.70	8.75	19.73	8.88

Table 3: Main results. ARC delivers clearest gains on in-domain tool use, especially on GRPO. Avg. is the mean over displayed metrics; red parentheses show Avg. change vs. backbone. TTFT (

\downarrow

) averages available

\tau

-bench latency measurements.

Method	Avg.	$\tau$ airline	$\tau$ retail	$\tau^{2}$ airline	$\tau^{2}$ retail	$\tau^{2}$ telecom
Qwen3-4B-noThink	22.38	20.00	22.32	21.40	26.31	21.90
Qwen3-4B-Think	29.54	27.51	37.10	28.07	29.21	25.83
GRPO (4B)	22.33	29.33	10.72	36.94	15.96	18.69
GRPO+ARC (4B)	34.23	29.33	40.00	32.00	39.77	30.03

Table 4: Qwen3-4B case study. ARC remains effective at smaller scale and outperforms no-think, think, and 4B GRPO baselines on the in-domain tool-use suite.

Figure 6: Training-time instruction removal underperforms the default ARC setup. Left: average number of off-strategy rollouts per group, where an rollout has a predicted strategy different from the injected strategy. Right: standard deviation of off-strategy counts across groups. Lower values are better in both panels, indicating more consistent within-strategy rollout grouping.

Training Setting	Avg.	$\tau$ Avg.	$\tau^{2}$ Avg.	Reasoning Avg.	IFBench	ArenaHard
No removal (ARC default)	29.59	47.00	38.22	31.58	15.99	15.18
Linear removal	27.28	38.21	31.83	34.12	16.33	15.92
Constant removal	26.97	34.97	32.99	33.21	18.71	15.01

Table 5: Curriculum learning over strategy-instruction removal does not improve ARC: no removal performs best, linear removal is intermediate, and constant removal is weakest.

Figure 7: Domain-instruction interaction analysis. Inference-time strategy hints show no uniformly superior deployment interface.

Strategy Segment	Avg.	$\tau$ Avg.	$\tau^{2}$ Avg.	Reasoning Avg.	IFBench	ArenaHard
Progress Update	21.74	23.62	22.36	34.01	16.33	14.27
Progress Update + Direct Answer	24.91	31.67	32.46	27.79	17.69	14.96
Progress Update + Direct Answer + Clarify First	25.60	31.04	28.97	35.21	16.67	16.13
Full Suite (4)	29.59	47.00	38.22	31.58	15.99	15.18

Table 6: Strategy scaling ablation. Incremental strategy scaling consistently drives overall performance gains: the full strategy suite performs best, while reasoning tasks peak with three strategies and agentic benchmarks show the most dramatic sensitivity to strategy diversity.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Reward Comparisons	Avg.	$\tau$ Avg.	$\tau^{2}$ Avg.	Reasoning Avg.	IFBench	ArenaHard
Partial tool + answer judge	24.90	30.92	29.16	33.19	14.97	16.24
Exact tool + answer judge	27.89	37.31	31.94	36.24	18.03	15.94
Penalized tool + gated judge	27.98	43.93	36.49	29.60	16.33	13.57

Appendix

Table 7: Reward-construction ablation. All variants include the format reward. Exact tool matching improves tool use, and the final gated design gives the strongest tool-centric operating point. Unless explicitly varied, all main RL baselines and ARC variants in the paper use the final row.

Judge Model	Avg.	$\tau$ Avg.	$\tau^{2}$ Avg.	Reasoning Avg.	IFBench	ArenaHard
Qwen3-235B-Instruct	27.98	43.93	36.49	29.60	16.33	13.57
Qwen3.5-27B	21.67	25.69	22.55	27.97	19.39	12.77
Qwen3.5-122B	25.02	35.22	30.70	29.27	16.67	13.27

Appendix

Table 8: Reward-model ablation for the same Qwen3-8B policy under the final reward construction used throughout the main experiments. Varying only the external judge shows that the 235B judge provides the strongest tool-use operating point.

Training Ingredients
Configuration	Avg.	$\tau$ Avg.	$\tau^{2}$ Avg.	Reasoning Avg.	IFBench	ArenaHard
GRPO	25.43	35.81	31.72	27.25	18.71	13.66
GRPO + Entropy	24.23	33.53	29.42	27.66	16.67	13.85
GRPO + Strategy-Conditioned Grouping	27.80	43.93	35.55	29.60	16.33	13.57
GRPO + Strategy-Conditioned Grouping + Entropy	29.59	47.00	38.22	31.58	15.99	15.18
Entropy Sensitivity (with Strategy-Conditioned Grouping)

Appendix

Table 9: Ablation study on ARC: Training settings and entropy hyperparameters. Strategy-conditioned grouping provides the largest single gain over base GRPO. While entropy regularization alone slightly decreases performance, combining it with strategy-conditioned grouping further improves the overall operating point. The best entropy value (0.001) yields the highest average score.

Method	$\hat{\sigma}^{2}_{\mathrm{inter}}$	$\hat{\sigma}^{2}_{\mathrm{intra}}$	Ratio (95% CI)
GRPO	0.0555	0.1331	0.417 [0.002, 3.015]
ARC	0.0030	0.0401	0.074 [0.002, 0.410]

Appendix

Table 10: Estimated realized inter- and intra-strategy reward variance on rollout data. Ratio denotes

\hat{\sigma}^{2}_{\mathrm{inter}}/\hat{\sigma}^{2}_{\mathrm{intra}}

, with 95% confidence intervals.

Method	Tool-use Avg.	$\tau$ - Airline	$\tau$ - Retail	$\tau^{2}$ - Airline	$\tau^{2}$ - Retail	$\tau^{2}$ - Telecom
GRPO	33.35	$31.33\pm 4.62$	$40.29\pm 4.94$	$36.67\pm 6.43$	$40.64\pm 5.97$	$17.84\pm 2.82$
GRPO + ARC	41.73	$44.00\pm 3.06$	$50.00\pm 1.33$	$48.00\pm 6.11$	$45.61\pm 0.86$	$21.05\pm 0.91$
GRPO + ARC, 50% corrupted	35.01	$\mathbf{34.00\pm 3.46}$	$\mathbf{42.90\pm 1.33}$	$\mathbf{37.16\pm 0.73}$	$\mathbf{42.11\pm 3.16}$	$\mathbf{18.87\pm 1.07}$

Appendix

Table 11: Sensitivity of ARC to strategy-label corruption. The corrupted setting replaces 50% of strategy instructions with a uniformly sampled alternative strategy during training. Results are based on a single training run (seed 1); values after

\pm

denote standard deviations over three independent evaluation runs.

Method	Tool-use Avg.	$\tau$ -Airline	$\tau$ -Retail	$\tau^{2}$ -Airline	$\tau^{2}$ -Retail	$\tau^{2}$ -Telecom
Qwen3-8B-NoThink	$22.16\pm 2.37$	$12.00\pm 0.00$	$29.86\pm 5.31$	$14.61\pm 7.79$	$36.55\pm 1.01$	$17.80\pm 2.53$
Qwen3-8B-Think	$31.35\pm 0.19$	$28.00\pm 6.00$	$36.81\pm 4.29$	$29.75\pm 3.27$	$38.71\pm 1.06$	$23.46\pm 4.19$
PPO	$33.60\pm 1.34$	$35.33\pm 3.06$	$41.45\pm 3.92$	$33.33\pm 3.06$	$38.89\pm 2.21$	$19.01\pm 1.34$
PPO + ARC	$\mathbf{38.45\pm 0.40}$	$\mathbf{39.33\pm 2.31}$	$\mathbf{46.09\pm 1.74}$	$\mathbf{41.61\pm 0.68}$	$\mathbf{44.44\pm 3.65}$	$\mathbf{20.76\pm 2.21}$
DAPO	$34.09\pm 3.26$	$31.33\pm 4.16$	$42.32\pm 5.52$	$35.33\pm 7.57$	$42.11\pm 4.88$	$19.37\pm 3.40$
DAPO + ARC	$\mathbf{34.24\pm 1.89}$	$\mathbf{34.67\pm 2.31}$	$\mathbf{41.74\pm 3.98}$	$\mathbf{34.50\pm 3.97}$	$\mathbf{40.64\pm 7.04}$	$\mathbf{19.64\pm 2.79}$

Appendix

Table 12: Evaluation uncertainty on the tool-use benchmarks. Results are reported as mean

\pm

standard deviation over

N=3

independent evaluation runs using the same trained checkpoint. Tool-use Avg. is the mean over the five displayed

\tau

-bench and

\tau^{2}

-bench metrics.

Method	Tool-use Avg.	$\tau$ -Airline	$\tau$ -Retail	$\tau^{2}$ -Airline	$\tau^{2}$ -Retail	$\tau^{2}$ -Telecom
PPO	$31.81\pm 1.87$	$33.78\pm 3.29$	$36.13\pm 5.09$	$32.89\pm 3.36$	$37.14\pm 1.92$	$19.10\pm 0.17$
PPO + ARC	$\mathbf{36.76\pm 1.56}$	$\mathbf{37.78\pm 1.39}$	$\mathbf{44.83\pm 2.96}$	$\mathbf{37.28\pm 3.78}$	$\mathbf{43.84\pm 0.59}$	$\mathbf{20.08\pm 1.18}$
DAPO	$34.69\pm 0.52$	$35.55\pm 4.02$	$40.97\pm 1.31$	$35.56\pm 1.02$	$41.91\pm 0.34$	$19.44\pm 0.46$
DAPO + ARC	$\mathbf{34.77\pm 0.70}$	$\mathbf{32.89\pm 1.68}$	$\mathbf{42.22\pm 3.36}$	$\mathbf{37.50\pm 2.62}$	$\mathbf{41.13\pm 0.85}$	$\mathbf{20.11\pm 0.58}$
GRPO	$34.74\pm 1.35$	$32.67\pm 1.34$	$42.61\pm 2.03$	$38.20\pm 1.37$	$41.80\pm 3.36$	$18.44\pm 0.80$
GRPO + ARC	$\mathbf{37.53\pm 1.53}$	$\mathbf{39.11\pm 1.68}$	$\mathbf{43.87\pm 4.36}$	$\mathbf{40.19\pm 1.65}$	$\mathbf{43.84\pm 1.60}$	$\mathbf{20.67\pm 1.53}$

Appendix

Table 13: Robustness to training randomness. Results are reported as mean

\pm

standard deviation over three independently trained models with different training seeds. Each seed-level score is obtained by averaging repeated evaluation runs for that checkpoint.

Parameter	Value
Base Model	Qwen3-8B
Learning Rate	1e-6
Batch Size	64
PPO Mini Batch Size	8
PPO Micro Batch Size (per GPU)	4
Max Prompt Length	6000

Appendix

Table 14: RL hyperparameters for the main experiments.

Explore similar work

Aug 18, 2026

Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce \textbf{F}eedback-\textbf{A}ware \textbf{C}redit \textbf{A}ssignment (\textsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, \textsc{FACA} improves the nine-domain

τ

-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

Yiwen Zhao, Zhihao Wen, Yuchen Mao +5

Fudan University · Ant International, Ant Group

RL for LLM Agents Conversational Agents RL for Tool Use Step-Level Credit Assignment in RL

Ask PDF

Feb 1, 2026

Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

Agentic Reinforcement Learning (ARL) trains large language models to interleave reasoning with external tool execution to solve complex tasks. Most existing ARL methods train a single set of parameters to support both reasoning and tool-use behaviors, implicitly assuming that joint training leads to improved overall agent performance. Despite its widespread adoption, this assumption has rarely been examined empirically. In this paper, we systematically examine this assumption by introducing Capability Effect Attribution (CEA), which provides quantitative evidence of interference between reasoning and tool-use behaviors. Through an in-depth analysis, we show that these two capabilities often induce misaligned gradient directions, leading to training interference that undermines the effectiveness of joint optimization and challenges the prevailing ARL paradigm. To address this issue, we propose Disentangled Action--Reasoning Tuning (DART), a simple and efficient framework that explicitly decouples parameter updates for reasoning and tool use via separate low-rank adaptation modules. With this simple change alone, DART outperforms all joint-optimization baselines and approaches the 2-Agent upper bound across thirteen benchmarks on retrieval-augmented QA and NL2SQL, further supporting our finding of capability interference under shared optimization.

Yu Li, Mingyang Yi, Xiuyu Li +6

School of Information, Renmin University of China, Beijing, China · Bytedance Inc., Beijing, China · Bytedance Inc., San Jose, USA

RL for Language Model Reasoning Gradient Interference Agentic Reasoning RL for LLM Agents RL for Tool Use

Ask PDF

Sep 1, 2026

ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning

Training open-ended agents via reinforcement learning (RL) is hindered by the lack of verifiable gold answers and scalable rubrics. Moreover, even near the model's capability boundary, long-horizon open-ended agentic tasks often yield brittle and unstable rewards, resulting in weak or noisy rollout contrast that obscures fine-grained optimization signals for group-based policy learning. To address these challenges, we propose ARISE-RL, a novel full-cycle self-evolution framework that couples a task/rubric Generator and a reasoning Solver through rubric-mediated co-evolution. The Generator grounds tool-related rubric criteria in real tool observations and is rewarded for producing valid, intermediate-difficulty tasks aligned with the Solver's evolving capability boundary. The Solver, in turn, learns from fine-grained rubric satisfaction signals through multi-step reasoning and tool use. We further introduce Reward-Gated Self-Evolution Distillation (RG-SED), which selectively distills a memory-augmented variant of the same policy back into itself only when the memory yields empirical reward improvement, thereby reducing distribution mismatch and avoiding blind imitation of noisy guidance. Finally, to support rigorous evaluation, we present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning. Extensive experiments demonstrate that ARISE-RL consistently achieves robust and stable overall state-of-the-art performance across all evaluated benchmarks.

Fanrui Zhang, Ruixue Ding, Qiang Zhang +13

Alibaba ATH Token Foundry · Hema, Alibaba Group

AI Agent Benchmarks Rubric-Based RL Reinforcement Learning with Verifiable Rewards RL for LLM Agents RL for Tool Use

Ask PDF