TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

Geng Chen; Ruotong Pan; Zhirui Yang; Qiqi He; Jiawei Chen; Zhang Yunfei; Chongyuan Chen; Minxuan Lv

cs.AISep 23, 2026

TRACER: Trajectory-Aligned Learning for Multi-Turn User Simulation

Authors: Geng Chen, Ruotong Pan, Zhirui Yang, Qiqi He, Jiawei Chen, Zhang Yunfei, Chongyuan Chen, Minxuan Lv, +4 more

Organizations: Peking University · Kuaishou Technology · Xi’an Jiaotong University · Institute of Software, Chinese Academy of Sciences · Institute of Information Engineering · Beijing University of Posts and Telecommunications

Abstract

Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. Yet current simulators often produce plausible individual responses without reproducing the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that models evolving user intent and aligns simulated trajectories with real ones. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly addressing reward sparsity and credit assignment challenges in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1 points, while outperforming all baselines on group-level conversion-rate error and semantic trajectory distance and generalizing to out-of-distribution scenarios. In human Turing tests, annotators identified TRACER conversations at near-chance accuracy. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion and response quality via simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.

Figures & tables

Figure 1: TRACER first learns from annotated human dialogues through supervised fine-tuning, then performs multi-turn reinforcement learning. A shared DTW alignment provides the trajectory cost for session-level rewards and local deviations for turn-level advantage modulation.

General Baselines
Method	Session-Level Behavior			Group-Level Fidelity		Semantic Content
Method	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
Qwen2.5-7B-Instruct	36.7	53.7	47.4	0.456	5.383	0.500
Qwen3-4B-Instruct	32.6	57.8	49.6	0.419	5.307	0.505
Gemini-3.1-Pro	40.8	56.4	52.6	0.425	1.588	0.480
Doubao-Seed-2.0-Pro	79.7	58.4	68.2	0.425	2.633	0.466

Table 1: Overall comparison of user simulators across three evaluation dimensions: Session-Level Behavior (PCR, ACC, F1), Group-Level Fidelity (Group-

\Delta

PCR, W1-Turns), and Semantic Content (OT-DTW). Groups are balanced 1:1; the session-level real PCR is 51.24%. Best results are in bold, second-best are underlined.

Figure 2: Human-indistinguishability evaluation. Accuracy is shown as bars, while the presence of non-human characteristics is indicated with markers. The dashed line at 50% represents the random guess threshold.

Method	Session-Level Behavior			Group-Level Fidelity		Semantic Content
Method	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
TRACER-7B	$50.3\pm 0.2$	$\mathbf{79.0}\pm 0.2$	$\mathbf{79.4}\pm 0.2$	$\mathbf{0.166}\pm 0.003$	$\mathbf{0.972}\pm 0.006$	$\mathbf{0.357}\pm 0.001$
$-$ w/o advantage modulation	$49.5\pm 0.4$	$\underline{78.7}\pm 0.1$	$\underline{78.8}\pm 0.2$	$\underline{0.181}\pm 0.004$	$\underline{1.030}\pm 0.014$	$0.369\pm 0.002$
$-$ w/o $R_{traj}$	$52.8\pm 0.3$	$77.4\pm 0.3$	$78.3\pm 0.3$	$0.187\pm 0.005$	$1.036\pm 0.020$	$0.373\pm 0.001$
$-$ w/o DTW alignment	$48.3\pm 0.3$	$77.7\pm 0.3$	$77.6\pm 0.4$	$0.186\pm 0.004$	$1.064\pm 0.014$	$0.367\pm 0.002$
$-$ w/o multi-turn RL	$52.8\pm 0.4$	$74.3\pm 0.3$	$75.3\pm 0.2$	$0.218\pm 0.004$	$1.081\pm 0.007$	$\underline{0.365}\pm 0.001$

Table 2: Ablation study of TRACER-7B on session-level behavior, dialogue structure, and semantic content. Results are reported as mean

\pm

standard deviation over five evaluation seeds.

Figure 3: Performance Comparison of Models on the Out-of-Domain Dataset.

Panel of Figure 4 — Figure 4: Overview of DM-Bench: User Profiles, Merchant Distribution, and Model Performance.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	$\lambda_{\mathrm{adv}}$	$\gamma$	$\rho$	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
Without advantage modulation	–	–	–	$49.54\pm 0.44$	$78.68\pm 0.13$	$78.85\pm 0.19$	$0.181\pm 0.004$	$1.030\pm 0.014$	$0.369\pm 0.002$
Default TRACER-7B	0.2	0.5	0.4	$50.29\pm 0.20$	$79.04\pm 0.19$	$79.36\pm 0.21$	$0.166\pm 0.003$	$\textbf{0.972}\pm 0.006$	$\mathbf{0.357}\pm 0.001$
$\rho=0.0$	0.2	0.5	0.0	$50.52\pm 0.32$	$78.36\pm 0.20$	$78.74\pm 0.15$	$0.176\pm 0.002$	$0.994\pm 0.012$	$0.373\pm 0.001$
$\rho=0.1$	0.2	0.5	0.1	$47.21\pm 0.22$	$79.06\pm 0.24$	$78.73\pm 0.26$	$0.164\pm 0.003$	$0.975\pm 0.007$	$0.369\pm 0.001$
$\rho=0.3$	0.2	0.5	0.3	$48.66\pm 0.19$	$78.55\pm 0.29$	$78.53\pm 0.28$	$0.170\pm 0.004$	$0.977\pm 0.017$	$0.365\pm 0.001$
$\rho=0.5$	0.2	0.5	0.5	$53.86\pm 0.16$	$78.42\pm 0.24$	$79.46\pm 0.23$	$0.177\pm 0.003$	$0.987\pm 0.009$	$0.365\pm 0.002$

Appendix

Table 3: Hyperparameter sensitivity of TRACER-7B. Each configuration is evaluated using its corresponding fixed checkpoint with five evaluation seeds (42–46); hyperparameter variants correspond to separately trained checkpoints. Results are mean

\pm

standard deviation across evaluation seeds, rather than independent training runs. PCR, ACC, and F1 are reported in percent. The reference PCR is 51.24%. Bold indicates the best mean for each directional metric.

Panel of Figure 5 — Figure 5: Training dynamics of TRACER-7B. We report the overall reward score, generation entropy, and mean response length during reinforcement learning. The solid curves show EMA-smoothed trends, while the faint curves denote the raw values.

Panel of Figure 6 — Figure 6: Training dynamics of TRACER-4B. We report the overall reward score, generation entropy, and mean response length during reinforcement learning. The solid curves show EMA-smoothed trends, while the faint curves denote the raw values.

Metric	Value	Notes
Inter-annotator agreement (Fleiss’ $\kappa$ )	0.77	Among human annotators
Agreement with human majority (Cohen’s $\kappa$ )	0.71	Claude vs. human majority
Raw agreement	87.0%	Claude vs. human majority
Gap in agreement ( $\Delta\kappa$ )	0.06	Within typical inter-annotator variability
LLM rationale plausibility $\geq 4$	92%	Rated on 5-point Likert scale

Appendix

Table 4: Evaluation of LLM-inferred subjective priors

Method	Session-Level Behavior			Group-Level Fidelity		Semantic Content
Method	PCR	ACC $\uparrow$	F1 $\uparrow$	Group- $\Delta$ PCR $\downarrow$	W1-Turns $\downarrow$	OT-DTW $\downarrow$
TRACER-4B	$51.6\pm 0.1$	$\mathbf{79.7\pm 0.2}$	$\mathbf{80.3\pm 0.2}$	$\mathbf{0.165\pm 0.004}$	$\mathbf{1.413\pm 0.017}$	$\mathbf{0.364\pm 0.002}$
$-$ w/o advantage modulation	$52.1\pm 0.2$	$\underline{78.5\pm 0.3}$	$\underline{79.2\pm 0.3}$	$\underline{0.178\pm 0.003}$	$\underline{1.415\pm 0.011}$	$0.373\pm 0.001$
$-$ w/o $R_{traj}$	$42.8\pm 0.2$	$75.8\pm 0.4$	$74.3\pm 0.4$	$0.201\pm 0.004$	$1.447\pm 0.033$	$0.371\pm 0.002$
$-$ w/o DTW alignment	$53.3\pm 0.1$	$77.6\pm 0.2$	$78.5\pm 0.2$	$0.187\pm 0.004$	$1.420\pm 0.011$	$0.367\pm 0.001$
$-$ w/o multi-turn RL	$39.1\pm 0.2$	$74.9\pm 0.4$	$72.2\pm 0.5$	$0.202\pm 0.003$	$1.758\pm 0.016$	$\underline{0.365\pm 0.002}$

Appendix

Table 5: Ablation study of TRACER-4B on session-level behavior, dialogue structure, and semantic content. Results are reported as mean

\pm

standard deviation over five evaluation seeds.

Linguistic Style
Dimension	Gemini	Doubao-pro	Doubao-char	SFT model	TRACER-7B
Lack of Human Imperfection	96.60%	98.80%	93.80%	12.20%	3.00%
Overly Written Register	4.00%	1.40%	1.20%	1.20%	4.00%
Overly Neat Structure	43.40%	50.60%	41.20%	29.60%	17.00%
Interaction Rhythm
Information Overload	87.20%	78.20%	78.40%	2.60%	0.80%

Appendix

Table 6: Fine-grained reasons why LLM-simulated users are judged as non-human. Lower percentages indicate better performance.

Figure 7: Prompt for the LLM-as-Judge evaluation is adapted from ( Zhu et al., 2026 ) .

Strategy	Decrease	Unchanged	Increase
Empathy	13/14 (92.9%)	4/7 (57.1%)	5/7 (71.4%)
Urgency	1/2 (50.0%)	1/1 (100.0%)	2/3 (66.7%)
Benefit / discount	14/15 (93.3%)	11/15 (73.3%)	16/24 (66.7%)
Informative help	55/61 (90.2%)	33/43 (76.7%)	49/59 (83.1%)
Contact request	37/43 (86.0%)	24/32 (75.0%)	54/62 (87.1%)
Probing / clarification	13/15 (86.7%)	16/22 (72.7%)	19/24 (79.2%)

Appendix

Table 7: Agreement between simulated and human-expected intent-change directions. Each cell reports matched / reference strategy instances (recall). The pooled row aggregates counts across strategy categories.

Model	CR (%) $\uparrow$	CT	RQ $\uparrow$
Model	CR (%) $\uparrow$	CT	Acc.	Help.	Under.	Coh.	Info.	Emp.	Overall
Claude-4.5-Sonnet	52.3	3.833	86.2	68.2	79.1	86.6	62.6	71.3	75.4
DeepSeek-V3.2	56.0	3.998	86.2	66.8	78.6	86.2	60.6	66.1	73.9
Doubao-seed-2.0-Pro	49.3	4.569	79.5	54.0	69.5	79.4	46.8	58.1	64.3
Gemini-3.1-Pro	51.4	3.878	86.4	67.9	79.5	86.1	61.0	69.9	74.8
Qwen3-235B-A22B-Instruct	55.7	3.968	82.6	60.9	74.0	83.0	54.0	62.9	69.3

Appendix

Table 8: Detailed results of mainstream LLMs on DM-Bench. Conversion Rate(CR) and Conversion Turns(CT) measure outcome-oriented performance, while Response Quality(RQ) is evaluated across multiple conversational dimensions.

Assistant	CR (%)	Rank
	Human Role-Play
DeepSeek-V3.2	47%	1
Qwen3-235B-A22B-Instruct	45%	2
Claude-4.5-Sonnet	45%	2
Gemini-3.1-Pro	40%	4
Doubao-seed-2.0-Pro	37%	5

Appendix

Table 9: Human role-play evaluation on 100 session contexts from DM-Bench. Each assistant is evaluated on the same user profiles, merchant information, and initial dialogue contexts. CR measures the proportion of valid role-play sessions ending in conversion.

RQ Dimension	Spearman $\rho$	p-value
Acc.	0.685	0.090
Help.	0.571	0.180
Under.	0.536	0.215
Coh.	0.679	0.094
Info.	0.393	0.383
Emp.	0.607	0.148

Appendix

Table 10: Spearman

\rho

correlation between Conversion Rate (CR) and Response Quality (RQ) metrics on DM-Bench.

Figure 8: Illustration of pointwise alignment and Dynamic Time Warping (DTW). In the left panel, strict pointwise alignment forces point $a$ to be matched with point $b$ according to their temporal indices, although the more plausible counterpart of $a$ is the locally shifted point $b^{\prime}$ . The right panel illustrates DTW-based nonlinear alignment, where one-to-one, one-to-many, and many-to-one correspondences are allowed under temporal-order constraints.

Explore similar work

Learning User Simulators with Turing Rewards
Jun 17, 2026Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu +5User Simulation Simulation-Based Reinforcement Learning
Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab
Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation
Aug 10, 2026Bo Wang, Ruixing Zhang, Yunqi Liu +4User Simulation Imitation
the State Key Laboratory of Complex and Critical Software Environment, Beihang University
SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
May 8, 2026Yada Pruksachatkun, Elaine Wan, Lyanna Chen +2User Simulation Multimodal Large Language Models
Salesforce Research · University of California Los Angeles

Jun 17, 2026, cs.CLJ/K move · Enter open · S save

Learning User Simulators with Turing Rewards

Yingshan Susan Wang, Cedegao E. Zhang, Linlu Qiu, Zexue He+4

Massachusetts Institute of Technology · Stanford University · MIT-IBM Watson AI Lab

User Simulation Simulation-Based Reinforcement Learning Icr-Rl

Open paper AskPDF arXiv

Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, research in the social sciences, and more. Existing approaches generally do so by training a large language model (LLM) to match a single ground truth response, either by maximizing the log probability or by using a similarity reward. We instead propose Turing-RL: a Turing-Test-based reinforcement learning approach for training user simulator models. Turing-RL uses a discriminative Turing reward with an LLM judge to score how indistinguishable a generated response is from the real user's given the user's history, and the user simulator LLM learns to produce responses indistinguishable from what the user could have said with such rewards. Across two different domains--conversational chat and Reddit forum discussion--we find that Turing-RL consistently outperforms baseline methods on both LLM and human evaluation metrics. Our study suggests that optimizing for indistinguishability, rather than response matching, is effective for learning user simulators.

Abstract

Explore similar work

Learning User Simulators with Turing Rewards

Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators