Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation

Atri Vivek Sharma; Brian Formento; Alessio Lomuscio

cs.LGOct 26, 2026

Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation

Authors: Atri Vivek Sharma, Brian Formento, Alessio Lomuscio

Organizations: Imperial College London · Safe Intelligence

Abstract

Multi-turn agents are commonly evaluated with LLM-based simulated users, which can be overly cooperative and can therefore overestimate an agent's performance. We here introduce StressSim, an intent-preserving adversarial user simulator framework for stress-testing agents with challenging yet valid user interactions. StressSim optimises the simulator's high-level adversarial plan for each task instance. Candidate planner prompts are evolved with a genetic prompt-optimisation loop that uses rollout traces, granular task-completion signals, and validity feedback to favour conversations that reduce task success without changing the user's underlying goal. We evaluate the approach on the MultiWOZ dataset and the retail and airline domains from tau-bench with three target LLM agents, comparing against standard simulators and hand-designed non-collaborative user behaviours. We demonstrate the effectiveness of StressSim in exposing robustness gaps across frontier LLMs, with tau-bench pass^4 scores decreasing by up to 69% on gemini-3.8-flash and 62% on gpt-5.6-sol. StressSim maintains goal preservation performance (which decreases by at most 10%), using an optimisation budget of 20 rollouts per task. Sensitivity analysis suggests that increasing the budget further decreases task success. Overall our results demonstrate that StressSim exposes robustness gaps missed by standard and hand-designed user simulators while maintaining conversation realism. This is achieved through lightweight prompt optimisation alone, without model fine-tuning.

Figures & tables

Figure 1: Adversarial user-simulator workflow. Task instructions initialise a goal-state tracker; an adversarial planner selects turn-level user policies; and a generator produces user utterances. Planner prompts are optimised to reduce task success while preserving goal alignment.

Dataset	User	$\mathbf{pass@1}$	$\mathbf{pass@4}$	$\mathbf{pass^4}$	IGA	$\mathbf{pass@1}$	$\mathbf{pass@4}$	$\mathbf{pass^4}$	IGA	$\mathbf{pass@1}$	$\mathbf{pass@4}$	$\mathbf{pass^4}$	IGA
		gpt-5.4-nano				kimi-k2.6				gemini-3.8-flash
MultiWOZ	standard	0.800	0.956	0.532	0.920	0.924	0.980	0.820	0.940	0.940	0.944	0.916	0.920
	NC emotional	0.812	0.980	0.512	0.880	0.912	0.980	0.760	0.960	0.920	0.960	0.840	0.880
	NC fragment	0.724	0.960	0.380	0.980	0.876	0.980	0.640	0.960	0.952	0.960	0.920	0.920
	NC tangential	0.772	0.960	0.448	0.920	0.900	0.980	0.780	0.960	0.940	0.980	0.860	0.860
	NC unavail.	0.416	0.820	0.044	0.800	0.604	0.952	0.208	0.840	0.652	0.960	0.160	0.980

Table 1: Results on MultiWOZ and the

\tau

-bench retail and airline datasets. We evaluate 50 tasks per benchmark with

n=5

trials per task and

k=4

. Initial Goal Alignment (IGA) is the fraction of trajectories judged as valid on the first attempt. “NC” refer to the non-collaborative hand-written baselines introduced by Shim et al. (2026) .

Panel of Figure 2 — Figure 2: Attack strength versus goal alignment, aggregated from Table 1 . Each point represents the mean of the evaluated metrics over the 9 dataset–LLM combinations; the $x$ axis is the drop in $\mathrm{pass^4}$ relative to the standard user, $(\mathrm{pass^4}_{\text{standard}}-\mathrm{pass^4}_{\text{sim}})/\mathrm{pass^4}_{\text{standard}}$ , and the $y$ axis is the mean IGA.

	Natural	Self-consist.	Progressive	Non-repet.	Goal-seeking	Goal-info. cov.	Final validity
Judge vs. Ann. 1	0.729	0.852	0.919	0.908	0.787	0.343	0.679
Judge vs. Ann. 2	0.968	1.000	0.948	1.000	0.928	0.736	0.840
Inter-annotator	0.768	0.852	0.921	0.908	0.812	0.525	0.839

Table 2: Cohen’s

\kappa

agreement on the human validation study of the automated validity judge over 100 sampled conversations. The first two rows report agreement between the automated judge and each human annotator; the last row reports inter-annotator agreement between the two human annotators.

Agent LLM	Optimised on	$\mathbf{pass@1}$	$\mathbf{pass@4}$	$\mathbf{pass^4}$	IGA
gemini-3.8-flash	none (standard)	0.588	0.788	0.320	0.960
	gpt-5.4-nano	0.456	0.668	0.184	0.880
	gemini-3.8-flash (self)	0.384	0.668	0.100	1.000
gpt-5.6-sol	none (standard)	0.480	0.704	0.296	0.960
	gpt-5.4-nano	0.488	0.736	0.264	0.940
	gemini-3.8-flash	0.524	0.696	0.304	0.920

Table 3: Transferability of the optimised planner prompts on

\tau

-bench retail. “Optimised on” gives the source agent the planner prompt was optimised against.

Strategy family	Description	MultiWOZ	$\tau$ -bench
Conditional and staged disclosure	Delays a fallback, constraint, identifier, or later goal until a specified condition or dialogue stage is reached.	0.48	0.34
Relational and cross-turn memory pressure	Expresses information through prior dialogue state rather than repeating the literal value.	0.50	0.00
Boundary, ranked, or relative option selection	Requires a boundary-optimal, ranked, non-default, or relationally described option.	0.36	0.00
Evidence and identifier-state pressure	Tests identifier roles and distinctions among availability, authorisation, and completed-action evidence.	0.24	0.08
Multi-item and multi-action coordination pressure	Requires mappings, grouping, or separation across several items, orders, or actions.	0.00	0.18
Consent and confirmation-stage intent pressure	Changes or withholds intent at the boundary between discussion and commitment.	0.06	0.14

Table 4: Adversarial strategy families discovered by the genetic optimiser, and their prevalence among optimised planner prompts per dataset. Prompts may exhibit multiple strategies.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Asset	Type / Source	License / Governing Terms
MultiWOZ	Dataset; official repository	MIT License
$\tau$ -bench	Dataset; official repository	MIT License
gpt-5.4- nano	Model; OpenRouter API (target agent) and OpenAI API (user simulator)	OpenRouter Terms of Service and OpenAI service terms
gpt-5.6- sol	Model; OpenRouter API	OpenRouter Terms of Service and OpenAI service terms
gemini- 3.8-flash	Model; OpenRouter API	OpenRouter Terms of Service and Google Gemini API Terms of Service
gpt-5.5	Model; OpenAI API (strategy taxonomy construction only)	OpenAI service terms

Appendix

Table 5: Licenses and governing terms for datasets and models used in the experiments.

Panel of Figure 3 — Figure 3: Sensitivity of the optimisation to its hyperparameters, with the other hyperparameter fixed at its default ( $\beta=5$ , 20 rollouts per task). (a) Increasing $\beta$ improves first-attempt goal alignment (IGA) but weakens adversarial pressure (higher passˆ4 ), while pass@4 is unaffected. (b) Increasing the rollout budget increases adversarial pressure, with varying impact on the different metrics, while IGA stays at 1.000 throughout.

Validity axis	Ann.	Acc.	Prec.	Rec.	F1	$\boldsymbol{\kappa}$	IAA $\boldsymbol{\kappa}$
		Judge vs. annotator
Natural	1	0.920	0.737	0.824	0.778	0.729	0.768
Natural	2	0.990	1.000	0.950	0.974	0.968	0.768
Self-consistent	1	0.990	1.000	0.750	0.857	0.852	0.852
Self-consistent	2	1.000	1.000	1.000	1.000	1.000	0.852
Progressive	1	0.970	0.920	0.958	0.939	0.919	0.921

Appendix

Table 6: Agreement between the automated validity judge and two human annotators on 100 sampled conversations, computed with invalid conversations as the positive class. The final column reports inter-annotator agreement (IAA) between the two human annotators.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training
Sep 29, 2026Ashish Jain, Armaan SandhuAI Agent Benchmarks User Simulation
Sarvam AI
Mind the Sim2Real Gap in User Simulation for Agentic Tasks
Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User Simulation Sim-to-Real Transfer
Carnegie Mellon University, Language Technologies Institute
RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
Apr 7, 2026Ming Zhu, Juntao Tan, Rithesh Murthy +6User Simulation Human Behavior Simulation
Salesforce AI Research

Sep 29, 2026J/K move · Enter open · S save

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Ashish Jain, Armaan Sandhu

Sarvam AI

AI Agent Benchmarks User Simulation

Open paper AskPDF arXiv

Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated user controls what information the agent receives and when, yet current benchmarks score only the agent and do not directly measure whether the user correctly executed its assigned role. We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success. Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks changes mean task reward by 15.2 points, while 24.4% of successful episodes contain a user-specification violation. The dominant failure is premature disclosure: users provide information before it is requested. This behavior has little effect on task reward, yet among successful episodes it causes the agent to make 1.06 fewer tool calls on average, changing the interaction being evaluated while preserving the reward. Finally, across seven proxies we identify an empirical cost-fidelity frontier, enabling practitioners to select the least expensive simulator that satisfies a required fidelity level.

Abstract

LLM Agent Evaluation

Explore similar work

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Mind the Sim2Real Gap in User Simulation for Agentic Tasks

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation