ARGUS: An Agenda-Driven Framework for Reliable User Simulation

Hang Zheng; Nan Jiang; Zichen Zhu; Zixu Yang; Zhiyang Tang; Shuai Fan; Lu Chen; Xiaobao Wu

cs.AIOct 15, 2026

ARGUS: An Agenda-Driven Framework for Reliable User Simulation

Authors: Hang Zheng, Nan Jiang, Zichen Zhu, Zixu Yang, Zhiyang Tang, Shuai Fan, Lu Chen, Xiaobao Wu,

Organizations: X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence SJTU AI Institute, Shanghai Jiao Tong University, Shanghai, China · Jiangsu Key Lab of Language Computing, Suzhou, China · AISpeech Co., Ltd., Suzhou, China · Suzhou Laboratory, Suzhou, China

Abstract

High-fidelity user simulators are essential for evaluating and training agents in interactive environments. However, current LLM-based simulators struggle with reliability issues, such as goal omission and policy non-compliance. We propose ARGUS (Agenda-dRiven Goal-oriented User Simulator), a neuro-symbolic framework that maintains the user's goal state using a dynamic agenda stack manipulated via symbolic operations. By rigorously decoupling state tracking from natural language generation, ARGUS facilitates strict goal progression and constraint-aligned interactions. For cost-efficient deployment, we distill ARGUS into compact models using a unified pipeline: evaluator-guided SFT followed by online RL with dense step-level rewards. To rigorously evaluate behavioral fidelity, we introduce a rubric-based metric Composite Behavioral Score (CBS). Evaluations on $τ^3$ -Bench and MultiWOZ demonstrate the significant superiority of the ARGUS architecture and the efficacy of our training pipeline. Code and data will be released soon.

Figures & tables

Figure 1: Stateless vs. stateful user simulation. Stateless generation may lose constraints or pending goals, while Argus preserves them through an explicit agenda stack.

Figure 2: Overview of ARGUS. (1) Agenda Initialization: structured user goals are parsed into a LIFO agenda stack with conditional behaviors. (2) Neuro-Symbolic User Simulation: a state controller updates the stack via symbolic operations ( Push , Pop , Modify , Wait ), and a response generator produces user utterances. (3) Two-Stage Training pipeline: teacher rollouts filtered by a turn-level evaluator (Soundness, Faithfulness, Adherence) yield SFT data, followed by online RL with step-level rewards.

Method	Retail		MultiWOZ		Airline
Method	CBS	TSR	CBS	TSR	CBS	TSR
Prompt-based
Vanilla (8B)	50.3	44.4	76.5	56.8	59.8	40.4
ARGUS (8B)	72.5	51.6	76.0	57.4	69.0	42.4
Vanilla (235B)	73.1	50.4	88.4	63.2	75.1	47.2
ARGUS (32B)	83.9	57.5	80.4	58.6	78.9	34.4

Table 2: CBS (%) and TSR (%) for trained simulators. Airline is the held-out OOD environment. Unless marked, all rows use Qwen3-8B. Prompt-based rows are included for comparisons.

Figure 5: Error rate (%) across 12 CBS subcriteria for six simulators on Retail. Subcriteria are grouped by their parent dimension. Darker cells indicate higher error rates.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Mean step-score bucket	$N$	Mean step score	Mean CBS (%)
$(0.2,0.3]$	8	0.2751	27.30
$(0.3,0.4]$	12	0.3584	46.70
$(0.4,0.5]$	30	0.4625	62.18
$(0.5,0.6]$	39	0.5618	70.86
$(0.6,0.7]$	51	0.6480	73.42
$(0.7,0.8]$	54	0.7552	82.37

Appendix

Table 3: Alignment between trajectory-aggregated step scores and final CBS. The 300 trajectories are grouped into eight non-empty score buckets; no trajectory falls in

[0,0.2]

Domain or layer	Supported operations
Hotel	Search by static attributes and dynamic constraints; check room availability and price; create, modify, cancel, and retrieve bookings.
Restaurant	Search by area, food type, price range, and reservation support; check table availability; create, modify, cancel, and retrieve reservations.
Train	Search by route, date, time, class, seat preference, and availability; assign seats; create, modify, cancel, and retrieve bookings.
Taxi	Search by route, time, capacity, vehicle attributes, and reservation support; estimate fares; create, modify, cancel, and retrieve taxi bookings.
Attraction	Search by area, type, name, and reservation support; check reservation feasibility; create, modify, cancel, and retrieve reservations when supported.
Geo and utility	Compute distances between places, retrieve nearby entities within a target domain, and perform simple arithmetic calculations.

Appendix

Table 4: Supported operations in the interactive MultiWOZ environment.

Dimension	Subcriterion	Description
D1: Faithfulness & Consistency	Hallucination	Simulator fabricates information in the response, introducing new details not present in the scenario or conversation facts, such as fictitious order numbers, account information, device status, or other business details.
	State Confusion	Simulator confuses the conversation state, mixing up attributes of different entities, different orders, different slots, or different check results, which leads to incorrect factual correspondences.
	Ungrounded Confirm	Simulator makes ungrounded confirmations, directly saying "yes" or "that’s right" without verification when the Agent proposes candidate information that is unprovided or incorrect.
	Cooperativeness	Simulator fails to cooperate truthfully, baselessly pretending not to know, shifting responsibility, or glossing over information that it already knows in the scenario and should have answered.
D2: Policy Adherence	Constraint Violation	Simulator violates constraints by prematurely leaking information it should not say, exposing meta-instructions, or accepting options and out-of-boundary solutions explicitly prohibited by the scenario.
	Cond. Logic Failure	Simulator fails to execute conditional logic correctly, meaning the if -> then rules in the scenario are either not triggered when they should be (missed trigger) or triggered when they should not be (false trigger).

Appendix

Table 5: The 12 binary subcriteria of the CBS evaluation framework, grouped by dimension.

Metric	Agreement
Cell CBS Pearson $r$	0.873
Cell CBS Spearman $\rho$	0.886
Cell CBS Kendall $\tau_{b}$	0.746
Protocol Failure agreement	93.81%
Protocol Failure Cohen’s $\kappa$	0.624
Method-pair ranking agreement	91.67%

Appendix

Table 6: Agreement between the Qwen and DeepSeek CBS judges on Retail domain. Cell-level correlations are computed over 24 method cells.

Dimension	CBS Subcriterion	Human IAA	Human–Qwen	Human–GPT	Qwen–GPT
Dimension	CBS Subcriterion	AC2	AC1	AC1	AC1
Faithfulness & Consistency	Hallucination	0.896	0.807	0.528	0.705
	State Confusion	1.000	0.846	1.000	0.901
	Ungrounded Confirm.	0.928	0.935	0.539	0.638
	Cooperativeness	0.939	0.929	0.916	0.887
Policy Adherence	Constraint Violation	0.828	0.603	0.701	0.739

Appendix

Table 7: Consolidated CBS reliability results across human inter-annotator agreement, human–LLM agreement, and Qwen–GPT model agreement. Human IAA uses Gwet’s AC2 because it is a multi-rater setting; pairwise comparisons use Gwet’s AC1.

Method	Retail		MultiWOZ
Method	CBS	TSR	CBS	TSR
Naive-SFT (trajectory)	62.3	47.0	84.5	64.0
Naive-SFT (step-level)	66.8	43.8	84.6	65.7
UGST-SFT (trajectory)	54.3	48.6	78.2	67.1
UGST-SFT (step-level)	62.6	47.2	81.4	64.9
Argus -SFT (trajectory)	81.9	51.9	84.5	63.6

Appendix

Table 9: Ablation of architecture and SFT data filtering. All variants use Qwen3-8B. “Trajectory” keeps samples only from task-successful trajectories, while “step-level” applies the evaluator in Appendix § B . For Naive-SFT and UGST-SFT, we design step evaluators tailored to their respective requirements.

Method	Qwen3-MoE-30B		Gemma3-27B
Method	CBS	TSR	CBS	TSR
Retail
Vanilla	65.1	44.6	68.1	32.6
+ Structured goal	76.4	48.4	71.6	39.1
+ Prompt constraints	68.8	47.5	67.6	43.7
+ Both	78.0	53.5	76.1	42.5

Appendix

Table 10: Input-representation and prompt controls for prompt-based simulators (%). “Structured goal” supplies the same parsed goal representation used by Argus ; “Prompt constraints” adds matched behavioral rules; “Both” applies both controls.

Method	Qwen3-MoE-30B		Gemma3-27B
Method	CBS	TSR	CBS	TSR
Retail
UGST (GPT-4o extractor)	60.7	52.3	74.7	41.8
UGST (GPT-5.1 extractor)	69.9	34.9	74.7	44.7
Argus	85.6	61.1	85.6	53.8
Airline

Appendix

Table 11: Control for the structured-goal extractor used by UGST (%). The original UGST setup uses GPT-4o, whereas the controlled variant uses GPT-5.1, matching the extractor model used by Argus .

Method	Retail		Airline		MultiWOZ
Method	CBS	TSR	CBS	TSR	CBS	TSR
Naive-SFT	53.7	43.3	59.6	32.0	69.6	53.2
UGST-SFT	49.7	46.1	57.5	31.6	69.4	59.6
UGST-RL	53.0	52.5	58.1	30.0	72.6	63.1
ARGUS-SFT	76.0	50.5	72.8	34.4	78.6	55.4
ARGUS-RL	82.6	57.7	80.4	35.6	83.0	58.3

Appendix

Table 12: Performance of trained Qwen3-4B user simulators.

Simulator	Easy		Medium		Hard		Expert		Overall
Simulator	TSR	CBS	TSR	CBS	TSR	CBS	TSR	CBS	TSR	CBS
Vanilla	77.1	70.7	47.6	68.8	38.2	71.9	35.0	69.9	48.8	70.3
React	72.5	68.4	54.4	69.6	42.5	67.6	41.0	66.3	52.2	68.0
Verify	72.3	72.1	45.8	71.2	42.3	73.8	36.6	67.2	48.6	71.0
Reflection	74.2	70.6	47.6	67.3	41.2	69.9	36.6	62.5	49.3	67.5
UGST	72.3	68.7	49.2	67.1	44.3	67.1	32.4	55.4	49.0	64.5

Appendix

Table 13: Performance of user simulators across task difficulty levels on the Retail domain and Qwen series. Complexity is defined as

n_{\text{cond}}+n_{\text{key}}

, where

n_{\text{cond}}

counts conditional behaviors and

n_{\text{key}}

counts database-modifying key actions ( cancel , modify , exchange , and return ). Bold indicates the best result per column.

Case 1: Skipping a Conditional Preference

Task and intended interaction. The user should first request a double room for two people in a centre guesthouse on January 8, 2026. If no centre guesthouse is available, the user may relax the type constraint and accept any centre accommodation. If multiple options remain, the user should request expensive properties. If there are still several expensive options, the user should ask the agent to choose the lowest total price among them.

Abridged dialogue. Agent: No centre guesthouse is available. Would you like me to search for other accommodation types? User: Okay, please look for any type of accommodation in the centre that can offer a double room for two people for one night. Agent: I found two centre hotels: Gonville Hotel (£147, expensive) and University Arms Hotel (£237, expensive). Would you like to proceed with one of these? User: Yes, please book the Gonville Hotel. It is the cheaper one, so that works for me.

Final-state result. The final booking is correct: the simulator books Gonville Hotel for the requested date, room type, party size, and stay length. The final-state check therefore marks the task as successful (TSR = 1.0).

CBS judgment. The judge preserves credit for faithfulness and format, since the simulator does not hallucinate unsupported facts and follows the output protocol. However, it removes credit for policy adherence and goal completion/progression. After the agent presents two options, the user immediately selects the cheaper hotel. The intended behavior is to first express the expensive-property preference and then use lowest price only as a tie-break among expensive options. Although the final hotel happens to be correct, the simulated user exposes the final choice through a shortcut rather than through the specified conditional goal progression.

Case 2: Skipping a Required Search Refinement

Task and intended interaction. The user should book standard-class train tickets for two people from Cambridge to Bishops Stortford on January 7, 2026. The initial request should only consider available trains departing after 23:30. If no train is found, the user should ask the agent to search again without the time restriction. If this produces too many results, the user should add a new constraint requiring departures after 05:28. Only then should the user choose the option with the earliest arrival time, using the lowest total price as a tie-break.

Appendix

Table 14: Representative shortcut cases where the final-state check succeeds, but CBS judgment identifies violations of the intended user-side interaction process. Dialogues are abridged for readability.

Method	Qwen3-8B		Qwen3-32B		Qwen3-MoE-30B		Qwen3-MoE-235B
Method	T	D	T	D	T	D	T	D
Vanilla	0.113	0.842	0.341	1.96	0.149	0.879	0.412	2.43
React	0.180	1.17	0.634	3.83	0.260	1.52	0.697	4.26
Verify	0.308	2.26	0.820	4.75	0.383	2.32	0.874	5.20
Reflection	0.317	2.68	0.848	4.88	0.362	2.15	0.956	5.71
UGST	1.09	9.79	3.88	29.5	2.00	11.8	5.00	26.0

Appendix

Table 15: Inference cost of prompt-based user simulators across model sizes. All values in

\times 10^{-3}

USD (1 CNY = $0.147); T = cost/turn, D = cost/dialogue; averaged over Retail and MultiWOZ.

Method	4B		8B
Method	T	D	T	D
Naive-SFT	0.066	0.418	0.098	0.609
UGST-SFT	0.079	0.452	0.127	0.752
UGST-RL	0.075	0.397	0.125	0.784
ARGUS-SFT	0.123	0.820	0.144	0.950
ARGUS-RL	0.088	0.661	0.138	0.974

Appendix

Table 16: Inference cost of trained user simulators. Same unit convention as Table 15 . All costs computed from actual token-usage records; models run locally via vLLM and priced at official API rates.

Figure 9: Per-dimension CBS comparison between SFT (at varying data scales) and RL. RL shows clear advantages on Faithfulness & Consistency and Policy Adherence, while SFT is stronger on Goal Completion. Format & Protocol is saturated for both.

Environment	SFT	RL	Test
Retail	100	406	114
MultiWOZ	196	508	155
Airline	–	–	50
Telecom	–	–	114

Appendix

Table 17: Benchmark split sizes by stage.

Task-Oriented Dialogue Systems

Task-oriented dialogue systems use conversation to understand user goals, gather information, and complete actions across multiple turns.

See all 59 papers

Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
May 13, 2026Harshita Chopra, Kshitish Ghate, Aylin Caliskan +3Human Behavior Simulation LLM Agent Evaluation
University of Washington, Seattle · Georgetown University, Washington DC
Towards User Simulators That Stay On Task And Still Sound Natural
Oct 22, 2026Mustafa Omer Gul, Philippe Laban, Tanya Goyal +1User Simulation Human Behavior Simulation
Cornell University · Microsoft Research

Mar 11, 2026J/K move · Enter open · S save

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie+7

Carnegie Mellon University, Language Technologies Institute

User Simulation Sim-to-Real Transfer LLM Agent Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the first study running the full

τ

-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

Abstract

Task-Oriented Dialogue Systems

Explore similar work

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

Towards User Simulators That Stay On Task And Still Sound Natural