DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

Xuyan Ye; Xiyan Fu; Shengda Fan; Jiawei Cui; Yankai Lin; Wei Lu

cs.LGOct 13, 2026

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

Authors: Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin, Wei Lu

Organizations: College of Computing and Data Science, Nanyang Technological University · Gaoling School of Artificial Intelligence, Renmin University of China

Abstract

Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps actor-training GPU-hours close to GRPO. Further experiments show improved sample efficiency and out-of-domain performance, and support the meter's ability to distinguish step quality and guide test-time action selection across actor sizes. The code is available at https://github.com/Xuyan923r/DeltaTTG.

Figures & tables

Figure 1: RLVR for agents faces sparse and delayed rewards. DeltaTTG learns a progress meter from diverse trajectories, measures each transition’s change in estimated remaining work, and uses it to redistribute the trajectory’s outcome credit.

Figure 2: Overview of the DeltaTTG framework. A TTG progress meter is trained on recorded agent trajectories through SFT and RL, using observed continuation outcomes and remaining-token labels. During actor training, the frozen meter estimates remaining completion effort; changes between successive states guide step-level redistribution of outcome-derived advantages, providing fine-grained process supervision.

Method	ALFWorld		PaperSearchQA		BFCLv3				Avg.
Method	Seen	Unseen	Original	Paraphrased	Non-Live	Live	Multi-Turn	Hall.	Avg.
Closed Models
GPT-5.6-Sol-low	$81.43_{\pm 1.75}$	$84.08_{\pm 0.93}$	$62.56_{\pm 0.17}$	$59.57_{\pm 0.38}$	$76.60_{\pm 0.32}$	$71.01_{\pm 0.24}$	$49.71_{\pm 2.72}$	$68.71_{\pm 0.59}$	70.11
Claude-Opus-5-low	$97.38_{\pm 0.34}$	$91.04_{\pm 0.00}$	$56.25_{\pm 0.08}$	$55.49_{\pm 0.18}$	$85.25$	$77.94$	$49.12$	$75.35$	74.00
Qwen3-4B
ReAct	$30.95_{\pm 1.47}$	$24.38_{\pm 2.46}$	$35.23_{\pm 0.56}$	$30.49_{\pm 0.69}$	$73.45_{\pm 0.28}$	$73.87_{\pm 0.37}$	$28.00_{\pm 3.53}$	$68.90_{\pm 0.35}$	40.53

Table 1: Performance comparison on ALFWorld, PaperSearchQA, and BFCLv3 (%,

\uparrow

). We report success rates on ALFWorld Seen/Unseen and PaperSearchQA Original/Paraphrased, and scores on the four BFCLv3 categories; Hall. denotes Hallucination/Irrelevance. For repeated evaluations, entries show the mean and population standard deviation over three test runs. Closed models provide reference results; within each open-backbone block, the highest and second-highest mean scores are bold and underlined , respectively. Avg. is the mean of the three benchmark averages (ALFWorld, PaperSearchQA, and BFCLv3), reported without a standard deviation.

Panel of Figure 3 — Figure 3: Sample efficiency on ALFWorld Seen using Qwen3-4B. (a) Success rate as a function of training steps. (b) Training batch success rate as a function of cumulative actor-generated response tokens, excluding prompt and progress-meter tokens. Solid curves show the EMA-smoothed training success rate, while dashed curves and shaded regions in (a) denote the mean test success rate and standard deviation across three evaluation seeds.

Variant	Pick	Clean	Heat	Cool	Look	Two	Avg.
Seen
Full DeltaTTG	97.14	96.30	89.58	86.67	94.87	83.33	91.67
$-$ TTG (+value)	97.14	85.19	81.25	76.00	76.92	87.50	85.71
$-$ TTG (+logit)	98.10	92.59	95.83	78.67	89.74	63.89	86.67
$-$ TTG (+feas.)	99.05	95.06	83.33	68.00	74.36	70.83	83.81
$-$ sign-aware reversal	95.24	92.59	85.42	78.67	66.67	62.50	82.38

Table 2: Ablation results on ALFWorld.

-

denotes removal; parentheses show replacement signals.

Figure 4: Actor-training compute on ALFWorld, including progress-meter inference for DeltaTTG . Token-normalized GPU-hours use GRPO’s 390.49M non-padding actor-training tokens, including prompts and responses, as the reference.

Panel of Figure 5 — Figure 5: Generalization analysis of DeltaTTG . (a)–(c) Out-of-domain generalization of Qwen3-4B policies trained on PaperSearchQA for (a) and AgentWorldModel for (b)–(c). Panel (a) uses a truncated y-axis starting at 4%, with an axis-break mark; panel (c) reports pass@8 on 89 tasks. (d) Transfer of an ALFWorld-trained progress meter to DeltaTTG optimization on PaperSearchQA, assessing progress signals beyond the meter’s training domain.

Supervision target	BFCL	GAIA	Hotpot QA	$\tau^{2}$ - bench	Overall
Validation
Future Score	53.04	70.89	44.12	63.89	60.22
Step-to-Go	57.39	59.49	52.94	53.47	55.91
Token-to-Go	65.22	60.76	52.94	61.81	61.83
Test
Future Score	55.47	60.96	47.66	53.57	54.86

Table 3: AgentProcessBench validation and test step-quality accuracy (%). Overall pools evaluated steps across datasets; the best Overall in each split is bold. See Appendix E.6 for the evaluation protocol.

Figure 6: ALFWorld test-time guidance results. Arrows connect Natural to Progress Meter Guided rollouts; labels show changes in success rate (percentage points) and average steps. White/gray rows denote Seen/Unseen. The protocol is in Appendix E.5 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	ALFWorld		PaperSearchQA		AgentWorldModel
Hyperparameter	Qwen	Llama	Qwen	Llama	Qwen	Llama
Common settings
Learning rate	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-6}$
Training batch size	16	16	512	512	16	16
Rollout group size $G$	8	8	5	5	16	16
PPO epochs	1	1	1	1	1	1

Appendix

Table 4: Training hyperparameters by dataset and actor backbone. Qwen and Llama denote Qwen3-4B and Llama-3.1-8B-Instruct.

Parameter	Terminal- Bench 2.0	$\tau^{2}$ -bench	BrowseComp- Plus
Temperature	0.6	0.6	0.7
Top- $p$	0.95	0.95	0.8
Top- $k$	20	20	20
Thinking	Enabled	Enabled	Enabled
Max output tokens	8,192	2,048	10,000
Context limit	32,768	32,768	40,960

Appendix

Table 5: Actor decoding and serving settings for out-of-domain evaluation. Output limits are per response; context limits are in tokens.

Trajectory source	Progress-meter task category
ALFWorld (train)	Text-based interaction
PaperSearchQA (train)	Information seeking
AgentWorldModel (train)	Complex engineering tasks

Appendix

Table 6: Training domains of the three progress meters. Each meter is reused for both actor backbones in the corresponding main experiments.

Table 7: Detailed performance comparison across different ALFWorld task types. We report success rates (

\uparrow

) on the Seen and Unseen splits. The #Tasks row denotes the number of evaluation instances in each category. Overall is the task-count-weighted average of the category success rates.

Method	Overall	Airline	Retail	Telecom
Base	27.34	18.00	42.11	16.67
GRPO	26.26	16.00	42.11	14.91
GiGPO	23.02	10.00	34.21	17.54
HGPO	28.78	22.00	43.86	16.67
DAPO	19.78	14.00	27.19	14.91
DeltaTTG	33.09	32.00	50.00	16.67

Appendix

Table 8: Out-of-domain success rates (%) on

\tau^{2}

-bench.

Panel of Figure 7 — Figure 7: Detailed training dynamics across the displayed benchmark–backbone settings. Each row corresponds to one benchmark–backbone combination, with performance plotted against training steps (left) and cumulative generated response tokens (right). Token counts exclude prompt tokens; M denotes millions. The AgentWorldModel rows report training batch success rates. Evaluation overlays, where shown, report means and standard deviations across three seeds.

Actor	Split	Natural rollout (1)		Random selection (4)		Progress meter (4)
Actor	Split	SR	Steps	SR	Steps	SR	Steps
1.7B	Seen	10.00	46.34	11.43	46.59	17.14	44.44
	Unseen	12.69	46.57	12.69	46.69	20.15	43.04
4B	Seen	27.86	41.21	31.43	40.11	44.29	35.29
	Unseen	25.37	41.78	26.87	41.92	43.28	35.72
8B	Seen	20.00	42.88	24.29	42.02	34.29	38.43

Appendix

Table 9: ALFWorld test-time action selection with Qwen3 actors. Natural rollout uses one candidate per decision; random selection and progress-meter guidance each use four. SR denotes success rate (%,

\uparrow

), and Steps denotes average interaction length (

\downarrow

). Bold marks the best result in each row.

Step	Detailed trajectory	Predicted Token-to-Go
Task: For the ’Indie Fridays: January 10 Live Showcase’ event, define three ticket tiers: a free tier named ’Free Viewer’ with price 0 and limit 500 that includes live-only access with no chat, a ’Standard’ tier at price 10.00 USD limited to 300 that includes live access and chat, and a ’VIP Backstage’ tier at price 35.00 USD limited to 50 that includes live access, chat, post-event replay rights, and access to a VIP-only Q&A segment.
0	Action: list_tools . Observation: Tool discovery returns 25 tools, including event and ticket-tier creation.	1,200
1	Action: mcp_tool_create_event_for_channel : channel=1; title=Indie Fridays: January 10 Live Showcase; description empty; start=2024-01-10T12:00:00; end=2024-01-10T14:00:00; timezone=UTC; privacy=public; is_public=true. Observation: Event created: id 7, channel 1; requested title; January 10, 2024, 12:00–14:00 UTC; public.	1,000
2	Action: mcp_tool_create_ticket_tier : event=7; name=Free Viewer; description empty; price=USD 0; limit=500; live=true; chat=false. Observation: Free Viewer tier created: id 11, event 7; USD 0, limit 500; live=true; chat, replay, VIP Q&A=false.	600
3	Action: mcp_tool_create_ticket_tier : event=7; name=Standard; description empty; price=USD 10; limit=300; live/chat=true. Observation: Standard tier created: id 12, event 7; USD 10, limit 300; live/chat=true; replay and VIP Q&A=false.	200
4	Action: mcp_tool_create_ticket_tier : event=7; name=VIP Backstage; description empty; price=USD 35; limit=50; live, chat, replay, VIP Q&A=true. Observation: VIP Backstage tier created: id 13, event 7; USD 35, limit 50; live, chat, replay, and VIP Q&A=true.	170

Appendix

Table 10: AgentWorldModel trajectory: complete six-step trace.

Step	Detailed trajectory	Predicted Token-to-Go
Task: Send one SMS via “Support Line” from +14155550123 to +447911123456: “Hi Alex, your order #4829 has shipped and should arrive by Friday.” Request a status callback to https://example.com/sms-status .
0	Action: list_tools . Observation: Tool discovery returns 32 tools, including account lookup and outbound messaging.	1,200
1	Action: mcp_tool_get_messaging_service_by_name_for_user : name=Support Line. Observation: Service SID: MG_SUPPORTLINE001 ; account ID=1; primary sender ID=1.	770
2	Action: mcp_tool_create_outbound_message : account SID= MG_ACCOUNT001 (assumed); service SID= MG_SUPPORTLINE001 ; sender, recipient, SMS body, and callback as requested. Observation: The tool returns "message": null .	980
3	Action: mcp_tool_create_outbound_message : repeats the preceding request with the same account SID. Observation: The tool again returns "message": null .	1,200
4	Action: mcp_tool_get_phone_number_by_e164 : number=+14155550123. Observation: Phone ID=1; account ID=1; SMS capability enabled.	Failure

Appendix

Table 11: AgentWorldModel trajectory: unsuccessful SMS task; complete eight-step trace.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 583 papers

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
May 20, 2026Kaiyi Zhang, Wei Wu, Yankai LinPolicy Gradient Token-Level Credit Assignment
Gaoling School of Artificial Intelligence, Renmin University of China · Ant International
Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
Aug 13, 2026Zechuan Wang, Siyuan Lu, Hongxuan Zhang +3Token-Level Credit Assignment Step-Level Credit Assignment in RL
Zhejiang University · AWorld Team, Inclusion AI · Shanghai Innovation Institute +2
A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
May 7, 2026Dingwei Chen, Zefang Zong, Zhipeng Ma +5Credit Assignment in RL Policy Gradient Methods
Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University

May 20, 2026J/K move · Enter open · S save

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

Kaiyi Zhang, Wei Wu, Yankai Lin

Gaoling School of Artificial Intelligence, Renmin University of China · Ant International

Policy Gradient Token-Level Credit Assignment RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose

\textbf{DelTA}

, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.

Abstract

RL for LLM Agents

Explore similar work

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

A2^22TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping