TRACER: Early Failure Detection for Task-Oriented Dialogue

Erfan Nourbakhsh; Rocky Slavin; Ke Yang; Anthony Rios

cs.CLJul 4, 2026

TRACER: Early Failure Detection for Task-Oriented Dialogue

Authors: Erfan Nourbakhsh, Rocky Slavin, Ke Yang, Anthony Rios

Organizations: The University of Texas at San Antonio

Abstract

Task-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has already gone wrong. We present TRACER, a method for early failure detection in task-oriented dialogue. TRACER predicts from a partial dialogue whether the full conversation will eventually fail by combining simple trajectory signals from belief-state changes with text representations of the evolving dialogue state. We evaluate the method in both oracle and generated belief-state settings, and test how well it works when only 25%, 50%, 75%, or 100% of the dialogue is visible. Across these settings, TRACER detects useful failure signals well before the end of the conversation and outperforms heuristic, classical, and single-stream baselines. These results suggest that early failure detection can provide a practical warning signal for dialogue systems before the interaction fully breaks down.

Explore similar work

When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories
Jun 3, 2026Avinash Baidya, Xinran Liang, Ruocheng Guo +2Early Failure Prediction Early Warning
ReacTOD: Bounded Neuro-Symbolic Agentic NLU for Zero-Shot Dialogue State Tracking
May 18, 2026Yanjun Lin, Zimo Xiao, Kartik Natarajan +7Neuro-Symbolic Framework Iterative Self-Correction
Wait! There's a Way Out: A Decision Mechanism for Forecasting Conversational Derailment
May 28, 2026Laerdon Kim, Vivian Nguyen, Cristian Danescu-Niculescu-MizilEarly Failure Prediction

Jun 3, 2026 · cs.CLJ/K move · Enter open · S save

When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories

Avinash Baidya, Xinran Liang, Ruocheng Guo, Xiang Gao+1

1Intuit AI Research · 2Princeton University

Early failure alerting requires deciding, while a dialog or agent trajectory is still unfolding, whether to flag it as likely to fail. This is challenging because supervision is typically available only as a trajectory-level success/failure label while alerts must be raised from partial interactions. Prior early-classification methods often bridge this gap by assigning the terminal label to every prefix, treating every turn as failure evidence. We hypothesize that this prefix-label assumption is poorly matched to multi-turn language interactions, where evidence of eventual failure is sparse and often delayed. In this paper, we introduce a two-stage approach that learns from this sparse evidence structure and uses the resulting risk estimates for controllable early alerting. Specifically, our attention-based failure predictor learns sparse turn-level failure evidence from trajectory labels and uses it to estimate failure risk from partial histories. We then pair this predictor with

α

-STOP, a single preference-conditioned stopping policy that selects an accuracy-earliness operating point at inference time rather than training a separate trigger for each preference. Across five benchmarks spanning customer support, task-oriented dialog, persuasion, tool use, and planning, we first show that high-relevance failure evidence occupies only 4.7-11.3% of turns and first appears after 59.0-83.6% of trajectories on average. We further show that the attention-based predictor improves Pareto-frontier quality (hypervolume) by 1-10% over naive prefix supervision, and that the full system improves frontier quality by 3-42% over state-of-the-art trigger policies while reducing training cost per operating point by 1-3 orders of magnitude.

Early Failure Prediction Early Warning Step-Level Supervision Early Layers

Open paper Ask about it PDF

Abstract

Explore similar work

When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories

ReacTOD: Bounded Neuro-Symbolic Agentic NLU for Zero-Shot Dialogue State Tracking

Wait! There's a Way Out: A Decision Mechanism for Forecasting Conversational Derailment