Cost-Aware Abstention for LLM-Based IT Ticket Classification

Sachin Gupta

cs.LGOct 22, 2026

Cost-Aware Abstention for LLM-Based IT Ticket Classification

Authors: Sachin Gupta

Organizations: Independent Researcher United States

Abstract

Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.

Figures & tables

Panel of Figure 1 — Fig. 1: Evaluation pipeline from ticket text to auto-routing or human triage, with prediction, confidence estimation, and abstention separated.

Property	Value
# tickets	47,837
# classes	8
Train / Calib. / Test	33,486 / 4,784 / 9,567
Median ticket length (chars)	175
Class imbalance (max/min)	$7.74\times$

TABLE I: Dataset summary (public IT Service Ticket Classification Dataset).

Baseline (on 1,000-ticket subsample)	Acc.	Macro-F1	ECE-raw
TF-IDF + LR	0.847	0.842	0.111
TF-IDF + Linear SVM	0.867	0.869	0.056
TF-IDF + Random Forest	0.804	0.791	0.172
DistilBERT (fine-tuned)	0.879	0.883	0.043

TABLE II: Non-LLM baselines on the LLM test subset (

n=1{,}000

Model	Setting	Acc.	Macro-F1	ECE r	ECE i	AUROC r	AUROC i	AURC r	AURC i
TF-IDF + LR	classical	0.847	0.842	0.107	—	—	—	—	—
TF-IDF + Linear SVM	classical	0.863	0.864	0.046	—	—	—	—	—
TF-IDF + Random Forest	classical	0.818	0.806	0.182	—	—	—	—	—
DistilBERT (fine-tuned)	classical	0.877	0.874	0.041	—	—	—	—	—
Claude Haiku 4.5	few-shot	0.477	0.459	0.386	0.035	0.664	0.659	0.417	0.430
GPT-4o-mini	few-shot	0.459	0.450	0.541	0.021	0.500	0.500	0.541	0.541

TABLE III: Classification and confidence results. Non-LLM: full test; LLM: common 1,000-ticket subset.

r/i

: raw/isotonic (fit on calibration); ECE and tie-group AURC

\downarrow

, AUROC

\uparrow

. Bold marks the best value per metric within each evaluation block.

Fig. 2: Class distribution of the 47,837-ticket corpus across the eight categories.

Model	Setting	ECE M9	ECE M8	AUROC M9	AUROC M8
Claude Haiku 4.5	few-shot	0.148	0.210	0.692	0.590
GPT-4o-mini	few-shot	0.205	0.239	0.707	0.593
Qwen2.5-7B-Instruct	few-shot	0.196	0.238	0.708	0.603
Llama-3-8B-Inst.-Lite	zero-shot	0.193	0.209	0.750	0.658
GPT-4o-mini	zero-shot	0.271	0.305	0.800	0.694
Claude Haiku 4.5	zero-shot	0.251	0.283	0.814	0.721

TABLE IV: Mixed and LLM-only agreement controls on the common 1,000-ticket test subset. The mixed

M{=}9

pool adds supervised TF-IDF + LR to the eight prompted conditions.

Policy	$\mathbb{E}[c]$	Reduction %	Cov. %	Acc. %
always_route	0.763	—	100.0	84.7
[email protected]	0.510	33.1	81.0	92.1
[email protected]	0.472	38.1	70.5	95.0
[email protected]	0.494	35.2	60.1	96.8
[email protected]	0.556	27.2	48.7	98.2
[email protected]	0.673	11.8	33.7	99.4

TABLE V: TF-IDF + LR test cost under

(0,1,5)

. Bold marks the calibration-selected operating point; reductions are relative to always_route .

Baseline	$\tau^{*}$	$\mathbb{E}[c]_{\text{a}}$	$\mathbb{E}[c]_{\tau^{*}}$	Reduction %	95 % CI	Wr. %
TF-IDF + LR	0.60	0.763	0.472	38.1	$[35.7,40.4]$	3.5
TF-IDF + Linear SVM	0.70	0.687	0.434	36.7	$[34.3,39.1]$	3.7
TF-IDF + Random Forest	0.60	0.910	0.571	37.3	$[35.0,39.5]$	2.1
DistilBERT (fine-tuned)	0.90	0.615	0.396	35.5	$[32.9,38.2]$	3.3

TABLE VI: Non-LLM test costs at calibration-selected thresholds under

(0,1,5)

\mathbb{E}[c]_{\mathrm{a}}

: always-route cost; Wr.: post-threshold wrong-route rate; ratio-bootstrap 95 % CIs.

$c_{\mathrm{triage}}\backslash c_{\mathrm{wrong}}$	2	5	10	20
0.5	[email protected] / 0.220 / 27.9%	[email protected] / 0.293 / 41.4%	[email protected] / 0.342 / 31.6%	[email protected] / 0.374 / 25.2%
1.0	[email protected] / 0.298 / 2.4%	[email protected] / 0.482 / 36.9%	[email protected] / 0.586 / 41.4%	[email protected] / 0.684 / 31.6%
1.5	[email protected] / 0.306 / $-0.2$ %	[email protected] / 0.606 / 20.6%	[email protected] / 0.789 / 47.4%	[email protected] / 0.934 / 37.7%
2.0	always_route † / 0.305 / 0.0%	[email protected] / 0.698 / 8.5%	[email protected] / 0.963 / 36.9%	[email protected] / 1.173 / 41.4%

TABLE VII: Post-hoc TF-IDF + LR cost-parameter sensitivity. For each setting, one policy is selected on calibration over always_route , always_defer , and thresholds 0.01–0.99, then evaluated once on test. Cells show policy / test

\mathbb{E}[c]

/ reduction relative to the lower-cost trivial test comparator. Negative reduction denotes higher cost. t@x abbreviates threshold@x ;

\dagger

marks the equal-cost boundary.

Score	$\tau^{*}$	$\mathbb{E}[c]$	Reduction %	95 % CI	Cov. %
Top probability	0.65	0.482	36.9	$[34.2,39.3]$	65.2
Top-two margin	0.43	0.467	38.8	$[36.5,41.1]$	68.5
Inverse entropy	0.44	0.507	33.6	$[30.8,36.2]$	68.3

TABLE VIII: TF-IDF + LR rejection scores under

(0,1,5)

. Thresholds are selected on calibration over a 0.01 grid and evaluated on test. Cost reductions and paired ratio-bootstrap 95 % CIs are relative to always-route.

Panel of Figure 11 — Fig. 3: TF-IDF + LR test cost over the predeclared threshold grid under $(0,1,5)$ . The calibration-selected $\tau=0.60$ yields a 38.1 % reduction (95 % CI $[35.7\,\%,40.4\,\%]$ ).

Panel of Figure 12 — Fig. 4: TF-IDF + LR coverage–accuracy curve. The marked calibration-selected point routes 70.5 % of test tickets at 95.0 % accuracy.

Setting	Prompt	Max tok.	Acc.	Macro-F1	Parse err.
zero-shot	v1	64	0.345	0.245	0.000
zero-shot	v2	128	0.250	0.197	0.000
few-shot	v1	64	0.245	0.271	0.185
few-shot	v2	128	0.320	0.289	0.000

TABLE IX: Joint prompt/budget sensitivity for Llama-3-8B on a 200-ticket subset. Template and token budget change together. Bold marks the higher accuracy within each zero-/few-shot pair.

Fig. 5: TF-IDF + LR confusion matrix without abstention on the full test set ( $n=9{,}567$ ). Color uses a square-root count scale; the three largest off-diagonal cells are labeled.

Selective Prediction

Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.

See all 287 papers

Apr 8, 2026

Triage: Routing Software Engineering Tasks to Cost-Effective LLM Tiers via Code Quality Signals

Context: AI coding agents route every task to a single frontier large language model (LLM), paying premium inference cost even when many tasks are routine. Objectives: We propose Triage, a framework that uses code health metrics -- indicators of software maintainability -- as a routing signal to assign each task to the cheapest model tier whose output passes the same verification gate as the expensive model. Methods: Triage defines three capability tiers (light, standard, heavy -- mirroring, e.g., Haiku, Sonnet, Opus) and routes tasks based on pre-computed code health sub-factors and task metadata. We design an evaluation comparing three routing policies on SWE-bench Lite (300 tasks across three model tiers): heuristic thresholds, a trained ML classifier, and a perfect-hindsight oracle. Results: We analytically derived two falsifiable conditions under which the tier-dependent asymmetry (medium LLMs benefit from clean code while frontier models do not) yields cost-effective routing: the light-tier pass rate on healthy code must exceed the inter-tier cost ratio, and code health must discriminate the required model tier with at least a small effect size (

\hat{p} \geq 0.56

). Conclusion: Triage transforms a diagnostic code quality metric into an actionable model-selection signal. We present a rigorous evaluation protocol to test the cost--quality trade-off and identify which code health sub-factors drive routing decisions.

Lech Madeyski

Wroclaw University of Science and Technology, Poland

LLM Routing Cost-Aware Inference Software Engineering

Ask PDF

Apr 16, 2026

TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification

Every call to an LLM classification endpoint produces a labeled input-output pair already retained in production logs. These pairs constitute a free, growing training set: a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost. The open questions are when the surrogate is reliable enough to deploy, what it handles versus defers, and how that boundary evolves as data accumulates. We introduce TRACER (Trace-based Adaptive Cost-Efficient Routing), an open-source system that trains ML surrogates on an LLM's own production traces and governs deployment through a parity gate: the surrogate is activated only when its agreement with the LLM exceeds a user-specified threshold α. To make the routing boundary transparent, TRACER generates interpretability artifacts describing which input regions the surrogate handles, where it plateaus, and why it defers. On a 77-class intent benchmark with a Sonnet 4.6 teacher, TRACER achieves 83-100% surrogate coverage depending on the quality target α; on a 150-class benchmark, the surrogate fully replaces the teacher. On a natural language inference task, the parity gate correctly refuses deployment because the embedding representation cannot support reliable separation. The system is available as open-source software.

Adam Rida

Independent Researcher.

Neural Surrogate Modeling LLM Inference Efficiency Classification LLM Routing

Ask PDF

Oct 21, 2026

Circuits of LLM Classifiers

Language models make it easy to build classifiers by simply prompting a model with the sample to classify. However, their predictions are neither deterministic nor uniformly reliable across samples. How can we combine predictions from different models to improve reliability? We view LLM classifiers as noisy measurement instruments of an underlying target label. Our goal is to maximize average classification accuracy at a given average query cost. To this end, we introduce circuits that organize LLM classifiers into an adaptive sequence of queries and decisions. We then characterize the optimal circuit, obtain the fundamental limits of achievable accuracy, and provide practical insights on realizing the optimal circuit. Across seven classification tasks spanning safety, health, political science, human preference, agent evaluation, decision-making and reasoning, our method improves the cost-accuracy frontier over routing and cascading baselines.

Hadi Khalaf, Sajani Vithana, Kunal Talreja +3

Harvard School of Engineering and Applied Science · Kempner Institute for the Study of Natural & Artificial Intelligence · School of Electrical and Computer Engineering, Georgia Institute of Technology +2

Large Language Model-Based Text Classification Language Model Ensembles Ensemble Learning Cost-Sensitive Learning

Ask PDF

Abstract

Selective Prediction

Explore similar work

Triage: Routing Software Engineering Tasks to Cost-Effective LLM Tiers via Code Quality Signals

TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification

Circuits of LLM Classifiers