TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

Geng Chen; Ruotong Pan; Zhirui Yang; Qiqi He; Jiawei Chen; Zhang Yunfei; Chongyuan Chen; Minxuan Lv

cs.AISep 23, 2026

TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

Authors: Geng Chen, Ruotong Pan, Zhirui Yang, Qiqi He, Jiawei Chen, Zhang Yunfei, Chongyuan Chen, Minxuan Lv, +4 more

Organizations: Peking University · Kuaishou Technology · Xi’an Jiaotong University · Institute of Software, Chinese Academy of Sciences · Institute of Information Engineering · Beijing University of Posts and Telecommunications

Abstract

Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. Yet current simulators often produce plausible individual responses without reproducing the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that models evolving user intent and aligns simulated trajectories with real ones. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly addressing reward sparsity and credit assignment challenges in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1 points, while outperforming all baselines on group-level conversion-rate error and semantic trajectory distance and generalizing to out-of-distribution scenarios. In human Turing tests, annotators identified TRACER conversations at near-chance accuracy. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion and response quality via simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.

Figures & tables

Figure 1: TRACER first learns from annotated human dialogues through supervised fine-tuning, then performs multi-turn reinforcement learning. A shared DTW alignment provides the trajectory cost for session-level rewards and local deviations for turn-level advantage modulation.

General Baselines
Method	Session-Level Behavior			Group-Level Fidelity		Semantic Content
Method	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
Qwen2.5-7B-Instruct	36.7	53.7	47.4	0.456	5.383	0.500
Qwen3-4B-Instruct	32.6	57.8	49.6	0.419	5.307	0.505
Gemini-3.1-Pro	40.8	56.4	52.6	0.425	1.588	0.480
Doubao-Seed-2.0-Pro	79.7	58.4	68.2	0.425	2.633	0.466

Table 1: Overall comparison of user simulators across three evaluation dimensions: Session-Level Behavior (PCR, ACC, F1), Group-Level Fidelity (Group-

\Delta

PCR, W1-Turns), and Semantic Content (OT-DTW). Groups are balanced 1:1; the session-level real PCR is 51.24%. Best results are in bold, second-best are underlined.

Figure 2: Human-indistinguishability evaluation. Accuracy is shown as bars, while the presence of non-human characteristics is indicated with markers. The dashed line at 50% represents the random guess threshold.

Method	Session-Level Behavior			Group-Level Fidelity		Semantic Content
Method	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
TRACER-7B	$50.3\pm 0.2$	$\mathbf{79.0}\pm 0.2$	$\mathbf{79.4}\pm 0.2$	$\mathbf{0.166}\pm 0.003$	$\mathbf{0.972}\pm 0.006$	$\mathbf{0.357}\pm 0.001$
$-$ w/o advantage modulation	$49.5\pm 0.4$	$\underline{78.7}\pm 0.1$	$\underline{78.8}\pm 0.2$	$\underline{0.181}\pm 0.004$	$\underline{1.030}\pm 0.014$	$0.369\pm 0.002$
$-$ w/o $R_{traj}$	$52.8\pm 0.3$	$77.4\pm 0.3$	$78.3\pm 0.3$	$0.187\pm 0.005$	$1.036\pm 0.020$	$0.373\pm 0.001$
$-$ w/o DTW alignment	$48.3\pm 0.3$	$77.7\pm 0.3$	$77.6\pm 0.4$	$0.186\pm 0.004$	$1.064\pm 0.014$	$0.367\pm 0.002$
$-$ w/o multi-turn RL	$52.8\pm 0.4$	$74.3\pm 0.3$	$75.3\pm 0.2$	$0.218\pm 0.004$	$1.081\pm 0.007$	$\underline{0.365}\pm 0.001$

Table 2: Ablation study of TRACER-7B on session-level behavior, dialogue structure, and semantic content. Results are reported as mean

\pm

standard deviation over five evaluation seeds.

Figure 3: Performance Comparison of Models on the Out-of-Domain Dataset.

Panel of Figure 4 — Figure 4: Overview of DM-Bench: User Profiles, Merchant Distribution, and Model Performance.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	$\lambda_{\mathrm{adv}}$	$\gamma$	$\rho$	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
Without advantage modulation	–	–	–	$49.54\pm 0.44$	$78.68\pm 0.13$	$78.85\pm 0.19$	$0.181\pm 0.004$	$1.030\pm 0.014$	$0.369\pm 0.002$
Default TRACER-7B	0.2	0.5	0.4	$50.29\pm 0.20$	$79.04\pm 0.19$	$79.36\pm 0.21$	$0.166\pm 0.003$	$\textbf{0.972}\pm 0.006$	$\mathbf{0.357}\pm 0.001$
$\rho=0.0$	0.2	0.5	0.0	$50.52\pm 0.32$	$78.36\pm 0.20$	$78.74\pm 0.15$	$0.176\pm 0.002$	$0.994\pm 0.012$	$0.373\pm 0.001$
$\rho=0.1$	0.2	0.5	0.1	$47.21\pm 0.22$	$79.06\pm 0.24$	$78.73\pm 0.26$	$0.164\pm 0.003$	$0.975\pm 0.007$	$0.369\pm 0.001$
$\rho=0.3$	0.2	0.5	0.3	$48.66\pm 0.19$	$78.55\pm 0.29$	$78.53\pm 0.28$	$0.170\pm 0.004$	$0.977\pm 0.017$	$0.365\pm 0.001$
$\rho=0.5$	0.2	0.5	0.5	$53.86\pm 0.16$	$78.42\pm 0.24$	$79.46\pm 0.23$	$0.177\pm 0.003$	$0.987\pm 0.009$	$0.365\pm 0.002$

Appendix

Table 3: Hyperparameter sensitivity of TRACER-7B. Each configuration is evaluated using its corresponding fixed checkpoint with five evaluation seeds (42–46); hyperparameter variants correspond to separately trained checkpoints. Results are mean

\pm

standard deviation across evaluation seeds, rather than independent training runs. PCR, ACC, and F1 are reported in percent. The reference PCR is 51.24%. Bold indicates the best mean for each directional metric.

Panel of Figure 5 — Figure 5: Training dynamics of TRACER-7B. We report the overall reward score, generation entropy, and mean response length during reinforcement learning. The solid curves show EMA-smoothed trends, while the faint curves denote the raw values.

Panel of Figure 6 — Figure 6: Training dynamics of TRACER-4B. We report the overall reward score, generation entropy, and mean response length during reinforcement learning. The solid curves show EMA-smoothed trends, while the faint curves denote the raw values.

Metric	Value	Notes
Inter-annotator agreement (Fleiss’ $\kappa$ )	0.77	Among human annotators
Agreement with human majority (Cohen’s $\kappa$ )	0.71	Claude vs. human majority
Raw agreement	87.0%	Claude vs. human majority
Gap in agreement ( $\Delta\kappa$ )	0.06	Within typical inter-annotator variability
LLM rationale plausibility $\geq 4$	92%	Rated on 5-point Likert scale

Appendix

Table 4: Evaluation of LLM-inferred subjective priors

Method	Session-Level Behavior			Group-Level Fidelity		Semantic Content
Method	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
TRACER-4B	$51.6\pm 0.1$	$\mathbf{79.7\pm 0.2}$	$\mathbf{80.3\pm 0.2}$	$\mathbf{0.165\pm 0.004}$	$\mathbf{1.413\pm 0.017}$	$\mathbf{0.364\pm 0.002}$
$-$ w/o advantage modulation	$52.1\pm 0.2$	$\underline{78.5\pm 0.3}$	$\underline{79.2\pm 0.3}$	$\underline{0.178\pm 0.003}$	$\underline{1.415\pm 0.011}$	$0.373\pm 0.001$
$-$ w/o $R_{traj}$	$42.8\pm 0.2$	$75.8\pm 0.4$	$74.3\pm 0.4$	$0.201\pm 0.004$	$1.447\pm 0.033$	$0.371\pm 0.002$
$-$ w/o DTW alignment	$53.3\pm 0.1$	$77.6\pm 0.2$	$78.5\pm 0.2$	$0.187\pm 0.004$	$1.420\pm 0.011$	$0.367\pm 0.001$
$-$ w/o multi-turn RL	$39.1\pm 0.2$	$74.9\pm 0.4$	$72.2\pm 0.5$	$0.202\pm 0.003$	$1.758\pm 0.016$	$\underline{0.365\pm 0.002}$

Appendix

Table 5: Ablation study of TRACER-4B on session-level behavior, dialogue structure, and semantic content. Results are reported as mean

\pm

standard deviation over five evaluation seeds.

Linguistic Style
Dimension	Gemini	Doubao-pro	Doubao-char	SFT model	TRACER-7B
Lack of Human Imperfection	96.60%	98.80%	93.80%	12.20%	3.00%
Overly Written Register	4.00%	1.40%	1.20%	1.20%	4.00%
Overly Neat Structure	43.40%	50.60%	41.20%	29.60%	17.00%
Interaction Rhythm
Information Overload	87.20%	78.20%	78.40%	2.60%	0.80%

Appendix

Table 6: Fine-grained reasons why LLM-simulated users are judged as non-human. Lower percentages indicate better performance.

Figure 7: Prompt for the LLM-as-Judge evaluation is adapted from ( Zhu et al., 2026 ) .

Strategy	Decrease	Unchanged	Increase
Empathy	13/14 (92.9%)	4/7 (57.1%)	5/7 (71.4%)
Urgency	1/2 (50.0%)	1/1 (100.0%)	2/3 (66.7%)
Benefit / discount	14/15 (93.3%)	11/15 (73.3%)	16/24 (66.7%)
Informative help	55/61 (90.2%)	33/43 (76.7%)	49/59 (83.1%)
Contact request	37/43 (86.0%)	24/32 (75.0%)	54/62 (87.1%)
Probing / clarification	13/15 (86.7%)	16/22 (72.7%)	19/24 (79.2%)

Appendix

Table 7: Agreement between simulated and human-expected intent-change directions. Each cell reports matched / reference strategy instances (recall). The pooled row aggregates counts across strategy categories.

Model	CR (%) $\uparrow$	CT	RQ $\uparrow$
Model	CR (%) $\uparrow$	CT	Acc.	Help.	Under.	Coh.	Info.	Emp.	Overall
Claude-4.5-Sonnet	52.3	3.833	86.2	68.2	79.1	86.6	62.6	71.3	75.4
DeepSeek-V3.2	56.0	3.998	86.2	66.8	78.6	86.2	60.6	66.1	73.9
Doubao-seed-2.0-Pro	49.3	4.569	79.5	54.0	69.5	79.4	46.8	58.1	64.3
Gemini-3.1-Pro	51.4	3.878	86.4	67.9	79.5	86.1	61.0	69.9	74.8
Qwen3-235B-A22B-Instruct	55.7	3.968	82.6	60.9	74.0	83.0	54.0	62.9	69.3

Appendix

Table 8: Detailed results of mainstream LLMs on DM-Bench. Conversion Rate(CR) and Conversion Turns(CT) measure outcome-oriented performance, while Response Quality(RQ) is evaluated across multiple conversational dimensions.

Assistant	CR (%)	Rank
	Human Role-Play
DeepSeek-V3.2	47%	1
Qwen3-235B-A22B-Instruct	45%	2
Claude-4.5-Sonnet	45%	2
Gemini-3.1-Pro	40%	4
Doubao-seed-2.0-Pro	37%	5

Appendix

Table 9: Human role-play evaluation on 100 session contexts from DM-Bench. Each assistant is evaluated on the same user profiles, merchant information, and initial dialogue contexts. CR measures the proportion of valid role-play sessions ending in conversion.

RQ Dimension	Spearman $\rho$	p-value
Acc.	0.685	0.090
Help.	0.571	0.180
Under.	0.536	0.215
Coh.	0.679	0.094
Info.	0.393	0.383
Emp.	0.607	0.148

Appendix

Table 10: Spearman

\rho

correlation between Conversion Rate (CR) and Response Quality (RQ) metrics on DM-Bench.

Figure 8: Illustration of pointwise alignment and Dynamic Time Warping (DTW). In the left panel, strict pointwise alignment forces point $a$ to be matched with point $b$ according to their temporal indices, although the more plausible counterpart of $a$ is the locally shifted point $b^{\prime}$ . The right panel illustrates DTW-based nonlinear alignment, where one-to-one, one-to-many, and many-to-one correspondences are allowed under temporal-order constraints.

Explore similar work

Jun 17, 2026cs.CL

Learning User Simulators with Turing Rewards

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu +5

Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab

User Simulation Simulation-Based Reinforcement Learning Icr-Rl

Ask PDF

Aug 10, 2026cs.CL

Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

User simulators are widely used as scalable environments for training and evaluating interactive assistants. Generating the next user turn is inherently one-to-many: the same profile and dialogue context may support multiple plausible continuations with different local interaction intents. A fluent response may therefore advance the dialogue through an inappropriate intent, such as acceptance rather than repair. Our key insight is that controllable user simulation should separate which local interaction intent the next user turn should realize from how that intent is expressed in language. We introduce UserIDA (User Intent-Directive Alignment), which exposes interaction intent as an explicit per-turn directive. UserIDA defines a six-way intent interface, learns directive-conditioned generation through supervised fine-tuning, and uses intent-calibrated policy optimization during group-based reinforcement learning. The reward preserves composite response quality while ensuring that intent-violating candidates rank below compliant alternatives in mixed groups. On LMSYS-USP, UserIDA achieves 86.6% intent accuracy, outperforming the strongest dedicated user-simulator baseline by 24.3 percentage points while improving semantic and stylistic similarity. In within-context interventions, it realizes at least four of the six target intents in 91.7% of evaluated dialogue states, compared with 22.9% for the strongest external baseline. These results establish per-turn intent control as a complementary dimension to response fidelity in user simulation.

Bo Wang, Ruixing Zhang, Yunqi Liu +4

the State Key Laboratory of Complex and Critical Software Environment, Beihang University

User Simulation Imitation Frictive Policy Optimization

Ask PDF

May 8, 2026cs.CL

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators

We present SalesSim, a framework and testbed for evaluating the ability of Multimodal Large Language Models (MLLMs) to simulate realistic, persona-driven customer behavior in multi-turn, multi-modal, tool-augmented online retail conversations. Unlike prior work that treat user simulation as surface-level dialogue generation, SalesSim models retail interaction and decision-making as a grounded, agentic process, where shoppers with diverse backgrounds, preferences, and dealbreakers interact with a sales agent, seek clarifications, and make informed purchasing decisions. For evaluation, we design a suite of metrics centered on decision alignment, measuring the consistency between the simulator's actions and its persona specifications, as well as conversational quality. We find several behavioral gaps after benchmarking 6 open and closed-source state-of-the-art models. First, while models produce fluent conversations, they display significantly lower lexical diversity and overdisclosure of criteria across personas compared to human conversations. Second, models tend to be persuaded by sales agent suggestions and drift from persona specifications. Even the strongest model achieves less than 79% average alignment with its underlying persona specifications. To make progress on these limitations, we propose UserGRPO, a multi-turn, multi-objective reinforcement learning recipe to optimize both conversational fluency and decision alignment under persona specifications. Our experiments demonstrate that UserGRPO boosts decision alignment of the baseline model by 13.8% while improving conversational quality. By introducing SalesSim, we provide a new testbed for the community to investigate and improve the adherence of user simulators in goal-oriented settings.

Yada Pruksachatkun, Elaine Wan, Lyanna Chen +2

Salesforce Research · University of California Los Angeles

User Simulation Multimodal Large Language Models Agentic Commerce

Ask PDF