IntentDelta: A Human-Constructed Benchmark for Evolving User Intent

Jieyu Zhou; Zijian Wang; Yun Wang

cs.CLOct 24, 2026

IntentDelta: A Human-Constructed Benchmark for Evolving User Intent

Authors: Jieyu Zhou, Zijian Wang, Yun Wang

Organizations: Georgia Institute of Technology, Atlanta, USA · Microsoft Research, Hong Kong SAR, China

Abstract

Language-model agents must adapt as users revise their requirements and priorities during complex tasks. We introduce IntentDelta, a human-constructed multi-turn benchmark with 638 trajectories and 3,766 turns across WebShop and TravelPlanner. Each turn pairs a user utterance with a cumulative intention state and human reference actions. The trajectories are cross-verified and cover feasibility- and preference-driven changes in constraints and priorities. We evaluate eleven agents on intention recovery and action quality. Average Intention Recovery is 33.1% in TravelPlanner and 44.7% in WebShop, while average Action Score is 31.6% and 41.0%, respectively. Agents struggle to preserve earlier requirements and distinguish soft preferences from must-have constraints. Correct intention recovery also does not guarantee successful execution, highlighting challenges in translating evolving user intentions into actions.

Figures & tables

Figure 1: Overview of evolving user intentions and agent evaluation. (a) User’s intention: The latent intention, hidden from the agent, evolves in response to real-world observations, including infeasibility and changes in preference. (b) Conversation: The user expresses the revised intention through subsequent utterances. In this example, the user relaxes the budget, demotes the nonstop requirement to a preference, and changes the preferred hotel location. (c) Agent evaluation: The agent is evaluated on both intention understanding and action outcomes.

Benchmark	Data scale	Multi- domain	Construction	Trade-off representation	Change conditions	Human action references
Frames El Asri et al. (2017)	1,369 dialogues	✗	Human	—	Infeasibility	✗
Flex-TravelPlanner Oh et al. (2025)	120 queries + 134 scenarios	✗	Synthetic	Hard/soft	—	NR
COMPASS Qin et al. (2025)	281 tasks	✗	LLM-generated	Hard constraints + utility	—	✗
Evolving User Intent Tack et al. (2026)	450 evaluated source tasks	✓	LLM-generated	—	—	✗
IntentDelta	638 trajectories 3,766 turns	✓	Human	Evolving priorities	Feasibility + preference	✓

Table 1: Benchmark comparison. Multi-domain denotes domain coverage; human action references denote per-turn human reference executions or plans. Dashes indicate no explicit priority/change-condition modeling; NR means not reported. Scale units are retained.

Characteristics	WebShop	TravelPlanner
Change Operations (% of events)
Add	20.4	37.3
Relax	18.0	8.3
Override	15.0	8.5
Reprioritize	46.6	46.0
Change Triggers (% of change turns)

Table 2: Intention change characteristics across WebShop and TravelPlanner. Operation percentages are calculated over change events, while trigger and compound-change percentages are calculated over turns containing intention changes.

Panel of Figure 2 — Figure 2: Distribution of trajectory lengths across WebShop and TravelPlanner, showing the percentage of trajectories with different numbers of turns.

	TravelPlanner				WebShop
	Intention		Action		Intention		Action
Model	Det. $\uparrow$	Rec. $\uparrow$	Hard Succ. $\uparrow$	Action Score $\uparrow$	Det. $\uparrow$	Rec. $\uparrow$	Hard Succ. $\uparrow$	Action Score $\uparrow$
gpt-5.6-sol	70.4	38.4	52.3	49.6	92.2	50.6	58.6	49.0
gpt-5.6-luna	72.1	38.2	46.5	42.3	91.6	49.4	54.6	45.1
claude-opus-4.8	68.4	37.3	47.0	42.3	89.2	52.0	52.0	41.9
claude-sonnet-5	70.8	39.1	55.6	51.6	90.6	50.6	52.0	42.7

Table 3: Main results on TravelPlanner and WebShop. Det. and Rec. denote Detection and Intention Recovery, respectively. All values are on a 0–100 scale; higher is better. Bold indicates the highest numerical value in each column.

Panel of Figure 3 — Figure 3: Intention recovery of carried-over constraints (not revised in the current turn) by how many turns ago they were introduced. Macro-average over 11 models; the decline holds for all 11 models in both domains.

Figure 4: Priority prediction in (a) TravelPlanner and (b) WebShop. Each row shows the distribution of predicted priorities for a gold priority, normalized to percentages. Diagonal entries indicate correct predictions.

Figure 5: Action Score for turns with fully versus incompletely recovered user intentions in TravelPlanner and WebShop. Full intention recovery is associated with higher Action Scores in both domains.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: WebShop annotation interface, illustrated with a ready-to-eat meal task. (a) Annotators edit user utterances, cumulative constraints, and their priorities. (b) Annotators inspect product information, available options, and supporting evidence before selecting a reference product.

Panel of Figure 7 — Figure 7: TravelPlanner annotation interface. (a) Annotators record a location-specific update allowing up to four billed accommodation nights in Denver while retaining a two-night limit in Colorado Springs, alongside other active constraints and their priorities. (b) Annotators edit a daily reference itinerary and inspect itemized costs and the total trip cost, illustrated with a Cleveland itinerary. The two panels show different trajectories.

Violation category	Full intention recovery	Incomplete recovery
Travel Planner
Budget	79.4%	39.6%
Preferences	10.3%	39.7%
Itinerary structure	3.8%	29.5%
Activity	0.0%	1.3%
Invalid booking (min. stay / capacity)	33.1%	50.4%

Appendix

Table 4: Constraint violations and invalid bookings in action failures, conditioned on intention recovery.

Scoring scheme	TravelPlanner	WebShop
	$\tau$ / $\Delta r_{\max}$	$\tau$ / $\Delta r_{\max}$
Linear 1:1	1.000 / 0	1.000 / 0
Linear 3:1	1.000 / 0	1.000 / 0
Linear 5:1	0.964 / 1	1.000 / 0
Lexicographic	0.964 / 1	1.000 / 0
Min. tier ratio	0.964 / 1	0.964 / 1

Appendix

Table 5: Sensitivity of model rankings to alternative soft-constraint scoring schemes. Kendall’s

\tau

measures agreement with the original 2:1 ranking;

\Delta r_{\max}

denotes the maximum absolute rank change across eleven models.

Domain	Precision $\uparrow$	Obsolete retention $\downarrow$
TravelPlanner	54.4	14.0
WebShop	84.5	13.8

Appendix

Table 6: Supplementary intention diagnostics (%), averaged across eleven models.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents
Sep 26, 2026Yanjie Zhang, Bowen Cao, Zixin Chen +1LLM Agents LLM Agent Evaluation
HKUST, Hong Kong SAR, China · CUHK, Hong Kong SAR, China · LIGHTSPEED, Shenzhen, China
You Changed Your Mind, The Model Didn't: Demystifying Intent in Multi-Turn Dialogue
Oct 5, 2026Junle Chen, Wei Chen, Zhengjun Huang +5Intent Classification Instruction Following
HKUST · Tencent
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Apr 6, 2026Rongqian Chen, Yu Li, Zeyu Fang +3Computer-Use Agent Evaluation Reward Modeling
George Washington University, Washington D.C., USA

Sep 26, 2026J/K move · Enter open · S save

When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents

Yanjie Zhang, Bowen Cao, Zixin Chen, Yushi Sun

HKUST, Hong Kong SAR, China · CUHK, Hong Kong SAR, China · LIGHTSPEED, Shenzhen, China

LLM Agents LLM Agent Evaluation LLM Agent Reliability AI Agent Benchmarks

Open paper AskPDF arXiv

LLM agents often operate over multi-turn interactions in which user intent changes before execution. We study intent drift: the failure mode in which superseded parts of the user's intent continue to influence the final answer or tool action. We introduce IntentFlux, an executable benchmark that converts verifiable tasks into dialogues with controlled intent changes while preserving their original graders. In a 627-case calibration, mean task score falls from 0.476 to 0.384 as dialogues contain more superseded and withdrawn information. Across eight models, the rate of fully correct solutions is significantly lower when the same final task must be recovered from an evolving dialogue rather than given directly in a single turn. We further introduce StateForge, which explicitly maintains the active requirements before generation. On General-Test, it improves mean task score from 0.367 to 0.467. Providing the ground-truth final state improves performance further but still does not recover single-turn performance, indicating that state-estimation errors explain only part of the gap. These results establish intent drift as a measurable multi-turn failure mode and explicit state maintenance as a partial mitigation.

Abstract

LLM Agent Evaluation

Explore similar work

When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents

You Changed Your Mind, The Model Didn't: Demystifying Intent in Multi-Turn Dialogue

IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents