cs.CLSep 29, 2026
SaveBeyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent
Organizations: ByteDance Inc., USA · University of Notre Dame
Abstract
Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent. However, this oracle communication assumption rarely holds in practice: users may miscommunicate, change their goals, and run out of patience. We define this task setting as Interactive Intent Alignment, where agents must recover and continuously track the user's current intent despite imperfect communication and evolving goals. To study this setting, we introduce Drift-Bench++, a principled benchmark construction pipeline for verified executable tasks with controlled misalignment and intent shifts, along with an interaction protocol featuring finite patience, diverse simulated users, and silent interaction-conditioned shifts. We further develop GRIP, a comprehensive evaluation protocol covering task grounding, user realism, inquiry effectiveness, and adaptation to evolving intent. Across diverse environments, models, and interaction conditions, stronger interaction consistently helps but remains far from oracle performance; Validation on deployed ProdAgent sessions further shows that the modeled failures are prevalent and consequential in deployment. By providing a unified, executable benchmark for interactive intent alignment, Drift-Bench++ offers a foundation for evaluating and advancing agents under realistic communication and evolving intent.
Figures & tables
Figure 1 : Overview of Drift-Bench++. a) A user query may misalign with latent intent, and the agent’s response may trigger a silent intent shift. b) Persona-conditioned behavior shapes the interaction beyond prompting style, and c) GRIP evaluates the resulting alignment process.
Figure 2 : Overview of the Drift-Bench++ data synthesis pipeline. Host tasks are mapped into intent graphs, and converted into verified misaligned instances. Lightweight benchmark-specific adapters and native oracles allow the remaining construction to be shared across domains.
Table 2 : Main comparison under the default evaluation condition (DeepSeek-V4-Pro backbone, Rational persona, single-fault requests, and intent shifting enabled). Results report mean sd over three seeded runs. Best results in each column are bolded and second-best results are underlined.
Figure 3 : Comparison between Various LLM Backbones. The benefit of interaction is backbone-robust. We show success rate of three agents across five LLM backbones. Clarification beats No-Ask baseline on every backbone and benchmark, and Self-evolving leads in 14 of 15 cells.
Figure 6 : Persona profiles of the Self-evolving agent on WebShop. Fans: one metric each, one wedge per persona, avg/max below. Bars: persona identification and consistency, judged from user messages alone with user profile prompt hidden on purpose.
Figure 7 : ProdAgent validation on 16,596 real production agent sessions over a 10-week deployment. Left: Comparison of misaligned and matched clean sessions in terms of platform evaluation scores and average tool calls per session; and Right: fault-type and shift-type distributions under strict two-judge agreement against the extracted intents provided by the platform.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmarks | Tasks | Graphs | Rejection Rate | Samples |
| WebShop | 500 | 257 | 31% | 1,780 |
| -Retail | 114 | 98 | 47% | 555 |
| -Airline | 50 | 40 | 48% | 208 |
Table 4: Key statistics of the construction pipeline with the rejection rate of the three-step verifier and the resulting sample numbers.
Figure 8 : Communication-fault taxonomy for interactive intent misalignment. Drift-Bench++ organizes communication failures into four families: flaws of intention, premise, parameter, and expression , which are further instantiated as eleven operational fault types used in verified query perturbation. The hierarchy highlights where cooperative breakdown occurs, ranging from obscured goals and false assumptions to missing task parameters and ambiguous language.
| Disclosure | Patience & Feedback | Silent Intent Shifts | |||||
| Persona | Slots/Answer | Volunteer | Refuse | Patience | Hint | Placements | Suggestibility |
| Rational | 1 | 10% | 0% | 1.0 | 30% | 100% | 20% |
| Dependent | 2 | 35% | 0% | 1.2 | 60% | 66%, 33% | 80% |
| Avoidant | 1 | 0% | 50% | 0.8 | 10% | 100% | 10% |
| Intuitive | 1 | 20% | 10% | 0.9 | 30% | 100%, 50% | 50% |
| Spontaneous | 1 | 30% | 5% | 0.7 | 40% | 100%, 50% | 60% |
Table 5: Persona configurations. Disclosure : slots revealed per answered question, the probability of volunteering an unasked requirement, and the probability of refusing an otherwise-valid answer. Patience & feedback : the multiplier scales the base patience budget ; hint is the probability that a rejection names what is wrong rather than a bare refusal. Silent intent shifts : placements position each scheduled shift as a percentage of the persona’s own initial budget, crossed as patience falls to or below that point; each crossing fires with probability (the scheduled fire coin), and an on-topic clarification question can instead fire a shift early with the persona’s suggestibility, consuming the last remaining placement so the shift count never grows (Appendix B.5 ).
Figure 9 : Left: Aim-Recovery changes from Just Ask (circle) to Self-Evolving (arrowhead) across five backbones. Right: successful hidden-intent episodes decomposed into Earned (solid) and Inferred (hatched) outcomes. Together, the panels show that stronger interaction primarily improves intent recovery and often shifts success toward explicitly recovered requirements, while Aim alone does not guarantee better recovery. Results are averaged over three seeded runs.
Figure 10 : Reaction under composed faults. Median turns from the last relevant intent shift to acceptance as fault complexity increases. Successful interactive trajectories generally require more post-shift turns at higher complexity, revealing the interaction cost of recovering coupled intent errors. Mean sd over three seeds; axes are scaled per panel.
Figure 11 : Patience consumed by intent shifting. Each arrow shows the change in remaining patience from shifts disabled (open circle) to enabled (filled marker). Shifts consume additional interaction budget in every method and benchmark, with the largest drain on -Retail. Values are from Table 8 ; axes are scaled per panel.
Figure 12 : Persona rankings across GRIP dimensions. Each cell ranks the five personas within one benchmark and metric for Self-Evolving (1 = best; darker = higher rank). Rankings show a stable broad difficulty structure but metric-dependent reversals, particularly between Rational and Dependent users. Just Ask exhibits a similar overall pattern. Values are from Table 9 .
Figure 13 : Generation prompt for a single communication fault, instantiated with a real task from the released corpus. Each of the eleven fault types has its own card and a type-specific final rule; the writer never sees the target product or the ground-truth answer set. Every output must pass dual extraction and executable admission before it ships. Section labels are added only for presentation.
Figure 14 : Extraction prompts that recover the structured edit a generated query actually expresses. Each questionnaire is answered independently by two models from different families at temperature zero, and a candidate proceeds only when both passes agree on the fault family and its target slot. Section labels are added only for presentation.
Figure 15 : Composite generation prompt, shown for a drawn pair of faults. Every drawn fault is pre-assigned its own target requirement before writing, and all cards are issued in one pass so no later rewrite can silently repair an already-checked fault; when three or more faults include a whole-message fault, that fault is applied as a second transformation-only pass. The final text is verified per component. Section labels are added only for presentation.
Figure 16 : The simulated user. The model only voices replies: what may be revealed is computed by code (the [PRIVATE] tags), acceptance of a proposal is decided by the executable verifier rather than by the model, and whether a rejection carries a hint is a persona trait rolled per rejection. A spoken reaction that contradicts the executable verdict is regenerated once with the verdict as a constraint. Section labels are added only for presentation.
Figure 17 : The five persona bios conditioning the simulated user (abridged; inherited verbatim from Drift-Bench so persona-conditioned results remain comparable), and the behavioral parameters Drift-Bench++ adds: slots revealed per answered question, the probability of volunteering an unasked requirement, the probability of declining a valid reveal, the patience multiplier, the probability a rejection carries a hint, intent-shift placements on the patience axis, and suggestibility to agent-prompted shifts. A persona changes what is said, never what is true: acceptance is executable and persona-independent.
Figure 18 : The prompt base every evaluated method shares: a deliberately strategy-free action interface and a store manual describing the environment mechanics. The No-Ask baseline receives the same prompt with the user-message action removed from the interface entirely; the patience meter is shown only to methods whose design includes budget planning. Section labels are added only for presentation.
Figure 19 : Method-specific guidance of the two clarification-capable reference methods. Just Ask adds a single ask-early instruction to the shared base. Self-Evolving extends the budget-planner method with lessons it distills from its own finished episodes (Figure 20 ); every run starts with an empty book, nothing crosses runs, personas, seeds, or benchmarks, and once the playbook becomes non-empty, its lessons are available from the beginning of subsequent episodes. Section labels are added only for presentation.
Figure 20 : The reflection prompt that collects Self-Evolving’s lessons between generations of a run, shown with the shopping-domain nouns (a retail run reads "customer" for "shopper"; the instruction is otherwise identical). Episode digests are projected onto agent-observable signals and mechanically redacted before reflection, and the credit labels are computed by code, so the reflection judges strategy rather than reconstructing what happened. Section labels are added only for presentation.
Figure 21 : An illustrative -Airline episode, condensed; quotes are verbatim. The plural “reservations” is lexically two-way, hiding a second booking; a scheduled relaxation resolves it silently before the first exchange. The deeper difficulty is structural: under airline policy a basic-economy booking cannot simply be modified, so the true intent’s golden sequence is update-then-cancel — the user’s own answer (“changing, not canceling”) describes their goal, not the policy’s implementation, and the missing cancellation surfaces only through rejection feedback. The agent asks the policy-required cancellation reason, the user volunteers the rebooking expectation, and the resubmission is accepted. Thirteen turns, patience 12 4.
Figure 22 : An illustrative WebShop episode, condensed from the released run; quotes are verbatim. The vague opening hides the $120 ceiling; the agent grounds the task in the catalogue before asking one two-part question that both disambiguates “natural hair” and elicits the budget — and that question triggers a silent refinement. A mechanically rejected purchase (Buy Now pressed on a details page) costs 4 patience and draws a natural complaint; the agent recovers, a second scheduled shift relaxes the intent, and the resubmitted purchase is accepted against the intent as it stands at commitment.
Figure 23 : An illustrative -Retail interaction trace showing recovery under repeated silent intent shifts.