B-CARE: Blind Classwise Assessment of Relative Evidence for Stance Error Ranking

Meysam Sabbaghan; Doina Caragea

cs.CLOct 21, 2026

B-CARE: Blind Classwise Assessment of Relative Evidence for Stance Error Ranking

Authors: Meysam Sabbaghan, Doina Caragea

Organizations: Department of Computer Science, Kansas State University

Abstract

Stance classifiers make errors that can be costly to identify under limited human-review budgets. Prioritizing potentially incorrect predictions remains challenging, particularly across heterogeneous base models. We introduce B-CARE (Blind Classwise Assessment of Relative Evidence for Stance Error Ranking), a prediction-blind framework for identifying high-risk stance predictions. Given only a target and text, a fixed LLM verifier assigns ordinal evidence scores to all candidate stance labels without observing the base model's prediction or confidence. B-CARE constructs a comparative risk score by contrasting the strongest alternative-class evidence with the evidence supporting the predicted class. The resulting classwise evidence profile can be reused across multiple base models without training a separate error detector or calibrating risk scores. We evaluate B-CARE on four stance benchmarks and three heterogeneous base LLMs, covering 12 dataset-model settings and 5,206 unique test instances. Evaluation includes error-ranking discrimination (AUROC and AUPRC), selective prediction (E-AURC), and review effectiveness (ErrorCapture at fixed review budgets). B-CARE achieves a mean AUROC of 0.735 and a mean AUPRC of 0.524, capturing 41.06% of errors within the highest-risk 20% of predictions. Although B-CARE does not uniformly outperform strong confidence-based baselines, it provides a reusable, complementary evidence signal for prioritizing potentially incorrect stance predictions across heterogeneous models.

Figures & tables

Figure 1: Overview of B-CARE. An independent, prediction-blind LLM verifier assigns ordinal evidence scores to all candidate stance labels without observing the base prediction. B-CARE contrasts evidence supporting the predicted label with its strongest alternative to produce an ordinal error-risk score for review prioritization. The evidence profile can be reused across multiple base classifiers without additional verifier generations.

Figure 2: AUROC comparison across 12 dataset–model settings. B-CARE and 13 competing error-ranking methods are evaluated on four stance-detection benchmarks and three base models, treating incorrect predictions as positive. Cell values are shown to three decimal places; boldface identifies the highest unrounded AUROC within each setting. The rightmost column reports the mean across all 12 settings. B-CARE achieves the highest mean AUROC (0.735) among the 14 methods in this primary comparison and leads in five settings.

Method	AUPRC $\uparrow$	E-AURC $\downarrow$	EC@10% $\uparrow$	EC@20% $\uparrow$	EC@30% $\uparrow$
MSP	$0.500\pm 0.093$	$0.121\pm 0.049$	$20.33\pm 4.67$	$37.48\pm 7.10$	$51.26\pm 7.66$
$p(\mathrm{True})$	$0.468\pm 0.139$	$0.132\pm 0.056$	$18.53\pm 5.98$	$32.62\pm 9.24$	$46.16\pm 9.53$
Self-Verification	$0.429\pm 0.069$	$0.138\pm 0.050$	$16.62\pm 4.97$	$31.50\pm 6.42$	$44.41\pm 7.22$
Predictive Entropy	$0.503\pm 0.092$	$\mathbf{0.119}\pm 0.050$	$20.20\pm 4.54$	$37.45\pm 7.08$	$51.25\pm 7.58$
Label Entropy	$0.353\pm 0.096$	$0.218\pm 0.058$	$16.78\pm 4.21$	$27.92\pm 5.51$	$36.94\pm 4.82$
Variation Ratio	$0.353\pm 0.096$	$0.218\pm 0.058$	$16.78\pm 4.21$	$27.92\pm 5.51$	$36.94\pm 4.82$

Table 1: Complementary error-ranking performance (mean

\pm

sample standard deviation across 12 dataset–model settings). Higher AUPRC and EC and lower E-AURC are better. EC is expressed as a percentage and uses expected fractional allocation at budget-boundary ties. Bold indicates the best mean in each column.

Figure 3: Component ablation of the B-CARE score across 12 dataset–model settings. Higher AUROC indicates better error ranking.

Figure 4: Ordinal versus continuous evidence scoring across 12 dataset–model settings. Dots show AUROC differences (continuous minus ordinal); horizontal bars indicate unadjusted 95% paired bootstrap confidence intervals (10,000 resamples). Positive values favor continuous scoring, negative values favor ordinal scoring, and intervals excluding zero indicate differences at the unadjusted 95% level.

Figure 5: Effect of prediction blindness on error-ranking AUROC. Cells report the AUROC difference (PBCE minus PACE) with unadjusted 95% paired-bootstrap confidence intervals (10,000 resamples). Darker purple indicates larger gains; stars denote intervals excluding zero.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$\boldsymbol{N}$	Tgt.	F	A	N
SemEval	1249	5	304	715	230
Vaccines	1000	1	396	259	345
COVID	800	4	263	243	294
P-Stance	2157	3	1032	1125	–
Total	5206	13	1995	2342	869

Appendix

Table T1: Evaluation-set statistics.

N

denotes the number of unique test examples, and Tgt. denotes the number of targets. F, A, and N represent FAVOR, AGAINST, and NONE, respectively. P-Stance does not contain the NONE class.

Dataset	Target	Test
SemEval-2016	Atheism	220
	Climate Change	169
	Feminist Movement	285
	Hillary Clinton	295
	Legalization of Abortion	280
	Subtotal	1249

Appendix

Table T2: Target-wise distribution of test examples across the four evaluation benchmarks. VaccineStance contains a single target (Vaccines), whereas SemEval-2016, COVID-19-Stance, and P-Stance contain five, four, and three targets, respectively. Counts correspond to the evaluation sets used in B-CARE.

Control	Settings	AUROC	$\boldsymbol{\Delta}$ vs. B-CARE
Same-Model LD	12	0.5856	+0.1490
Llama LD	12	0.6704	+0.0642
Random Ranking	12	0.5008	+0.2338
$R_{\mathrm{share}}$	12	0.6940	+0.0406

Appendix

Table T3: Additional error-ranking controls across 12 dataset–model settings. AUROC values are macro-averaged equally across the 12 settings.

\Delta

denotes the difference between the overall B-CARE mean AUROC (0.7346) and the corresponding control mean AUROC. Random Ranking averages 50 fixed-seed runs.

Baseline	Win	Loss	NS
MSP	4	4	4
Predictive Entropy	4	4	4
$p(\mathrm{True})$	8	2	2
Self-Verification	9	1	2
Direct Verification	12	0	0
Prediction-Aware Evidence	7	0	5

Appendix

Table T4: Summary of 156 paired bootstrap AUROC comparisons across 12 dataset–model settings. Win and Loss indicate significant positive and negative differences, respectively; NS indicates a nonsignificant difference under unadjusted 95% confidence intervals.

Figure A1: Complementarity between B-CARE and Predictive Entropy (PE) across 12 dataset–model settings. Top: AUROC of B-CARE, PE, and the exploratory equal-weight hybrid. Bottom: Errors captured exclusively by B-CARE, exclusively by PE, or by both within separate top-20% review sets. Segment heights represent percentages of all errors; labels indicate error counts.

Dataset	Model	B-CARE	PE	Hybrid	$\boldsymbol{\Delta_{\mathrm{B}}}$	$\boldsymbol{\Delta_{\mathrm{PE}}}$
SemEval	Phi	0.797	0.612	0.739	$-0.058$	$+0.127$
	Mistral	0.849	0.663	0.804	$-0.045$	$+0.141$
	Gemma	0.738	0.669	0.733	$-0.005$	$+0.064$
Vaccines	Phi	0.698	0.726	0.742	$+0.044$	$+0.017$
	Mistral	0.672	0.718	0.723	$+0.051$	$+0.005$
	Gemma	0.724	0.710	0.750	$+0.026$	$+0.040$

Appendix

Table T5: Per-setting AUROC of B-CARE, Predictive Entropy (PE), and the exploratory fixed equal-weight hybrid.

\Delta_{\mathrm{B}}

and

\Delta_{\mathrm{PE}}

denote hybrid AUROC minus B-CARE and PE AUROC, respectively. Bold indicates the highest AUROC in each setting. Values are rounded to three decimals.

Dataset	Model	$\boldsymbol{k}$	Errors	B-only	PE-only	Both	Union (%)	Jaccard
SemEval	Phi	250	460	124	53	87	57.4	0.330
	Mistral	250	550	135	41	92	48.7	0.343
	Gemma	250	358	87	47	78	59.2	0.368
Vaccines	Phi	200	330	43	55	62	48.5	0.388
	Mistral	200	301	33	58	59	49.8	0.393
	Gemma	200	377	63	59	65	49.6	0.348

Appendix

Table T6: Overlap of actual errors captured by B-CARE and Predictive Entropy (PE) within separately selected top-20% review sets.

k

is the number of reviewed instances per method, and Errors is the total number of incorrect base-model predictions. B-only, PE-only, and Both are mutually exclusive error counts. Union is the percentage of all errors captured by either method; Jaccard measures overlap between the two captured-error sets. Macro means are unweighted across 12 settings.

Dataset	Model	B-CARE	$-S$	$A$	$\Delta_{-S}$	$\Delta_{A}$	95% CI ( $\Delta_{-S}$ )	95% CI ( $\Delta_{A}$ )
SemEval	Phi	0.7969	0.7823	0.7933	+0.0146	+0.0036	[0.0093, 0.0203]	[-0.0058, 0.0131]
	Mistral	0.8490	0.8372	0.8521	+0.0119	-0.0031	[0.0081, 0.0160]	[-0.0096, 0.0036]
	Gemma	0.7378	0.7191	0.7419	+0.0187	-0.0041	[0.0119, 0.0262]	[-0.0143, 0.0065]
Vaccines	Phi	0.6984	0.7011	0.6162	-0.0026	+0.0822	[-0.0103, 0.0052]	[0.0637, 0.1014]
	Mistral	0.6723	0.6711	0.6093	+0.0013	+0.0630	[-0.0059, 0.0088]	[0.0452, 0.0809]
	Gemma	0.7243	0.7228	0.6427	+0.0015	+0.0815	[-0.0066, 0.0095]	[0.0632, 0.0999]

Appendix

Table T7: Detailed component ablation across 12 dataset–model settings (AUROC).

\Delta_{-S}

and

\Delta_{A}

denote the AUROC differences between B-CARE and the support-only and alternative-only variants, respectively. Confidence intervals are based on 10,000 paired bootstrap resamples and are unadjusted for multiple comparisons. Bold indicates the highest AUROC per setting.

Dataset	Base	B-CARE	LD	$\boldsymbol{\Delta}$	95% CI
SemEval	Phi	0.7969	0.7346	+0.0622	[0.0424, 0.0805]
	Mistral	0.8490	0.8028	+0.0462	[0.0313, 0.0605]
	Gemma	0.7378	0.6816	+0.0562	[0.0339, 0.0776]
Vaccines	Phi	0.6984	0.5883	+0.1102	[0.0867, 0.1344]
	Mistral	0.6723	0.5901	+0.0822	[0.0610, 0.1034]
	Gemma	0.7243	0.6133	+0.1110	[0.0886, 0.1335]

Appendix

Table T8: AUROC comparison of graded B-CARE and Evidence-Label Disagreement across 12 dataset–model settings.

\Delta

denotes B-CARE minus LD. Confidence intervals are unadjusted 95% paired-bootstrap intervals (10,000 resamples).

Metric	$\boldsymbol{\Delta}$	Wins	CI $>0$
AUROC	+0.0803	12/12	12/12
AUPRC	+0.0813	12/12	12/12
Recall@10%	+0.0129	12/12	6/12
Recall@20%	+0.0250	11/12	5/12
Recall@30%	+0.0591	12/12	12/12

Appendix

Table T9: Macro-average gains of B-CARE over LD. Wins count settings with positive point-estimate differences; CI

>0

counts unadjusted 95% paired-bootstrap intervals excluding zero in favor of B-CARE.

Figure A2: Macro-average improvement in expected error recall (percentage points) for B-CARE over Evidence-Label Disagreement at fixed review budgets. Results are averaged equally across 12 settings: SemEval, Vaccines, COVID-19, and P-Stance, each evaluated with Phi, Mistral, and Gemma. At a 50% review budget, mean recall is 0.7426 for B-CARE versus 0.6354 for LD; this comparison is descriptive.

Dataset	Model	B-CARE	LP	$\boldsymbol{\Delta}$
SemEval	Phi	0.794	0.786	+0.008
	Mistral	0.845	0.840	+0.005
	Gemma	0.736	0.728	+0.008
Vaccines	Phi	0.719	0.692	+0.027
	Mistral	0.702	0.671	+0.031
	Gemma	0.757	0.718	+0.039

Appendix

Table T10: Matched RunPod AUROC comparison between B-CARE and LlamaProb (LP).

\Delta=\mathrm{B\text{-}CARE}-\mathrm{LP}

; positive values favor B-CARE. Best AUROC values are shown in bold. Values are displayed to three decimal places.

Figure A3: AUROC differences between B-CARE and LlamaProb across four datasets and three frozen base models. Positive differences favor B-CARE; negative differences favor LlamaProb. Blue intensity represents the absolute magnitude of the difference, not statistical significance.

Model	$\boldsymbol{\Delta}$ AUROC	95% CI	Tie share
Phi	-0.095	$[-0.131,-0.061]$	72.5%
Mistral	-0.094	$[-0.129,-0.060]$	68.3%
Gemma	-0.097	$[-0.130,-0.065]$	66.5%

Appendix

Table T11: COVID diagnostic results from the matched RunPod evaluation (

n=800

\Delta

AUROC is B-CARE minus LlamaProb. Confidence intervals are estimated from 10,000 paired bootstrap resamples. Tie share is the percentage of the observed AUROC gap attributable to error–correct pairs tied by B-CARE. The Gemma difference is calculated from unrounded scores and may differ from subtraction of displayed three-decimal AUROCs.

Figure A4: Mean error capture across twelve matched dataset–model settings at review budgets of 10%, 20%, and 30%. B-CARE is shown in purple and LlamaProb in orange. Higher values indicate better error capture.

Metric	B-CARE	LlamaProb
Mean AUROC $\uparrow$	0.760	0.761
Mean AUPRC $\uparrow$	0.603	0.592
Mean E-AURC $\downarrow$	0.116	0.108
EC@10% $\uparrow$	26.1%	25.6%
EC@20% $\uparrow$	44.4%	42.8%
EC@30% $\uparrow$	58.5%	56.0%

Appendix

Table T12: Aggregate matched RunPod metrics averaged across twelve dataset–model settings. Arrows indicate the preferred direction. EC denotes error capture at the specified review budget.

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 91 papers

Oct 9, 2026

MoPrO-SD: Modular Prompt Optimization for Stance Detection

Stance detection remains challenging for large language models (LLMs) due to implicit opinions, sarcasm, and context-dependent expressions of stance. Although automatic prompt optimization offers an alternative to fine-tuning without updating model parameters, existing structured approaches are not specifically designed around stance-label semantics and class-specific ambiguities. We introduce MoPrO-SD, a stance-specific prompt optimization framework that decomposes prompts into functional semantic modules and refines them sequentially using LLM-based feedback. We study three refinement strategies: instruction-only optimization (I), instruction with reasoning (I+R), and instruction with per-example reasoning (I+R+E), using class-conditioned errors for stance-label refinement. We also introduce VaccineStance, a new human-annotated benchmark for vaccine stance detection. Experiments across four datasets and five open-weight LLMs show that MoPrO-SD improves upon the human-written Base prompt in many settings. On SemEval-2016, class-conditioned feedback achieves higher mean Macro-F1 than pooled feedback for four of five backbones. Both reasoning-based strategies, I+R and I+R+E, achieve strong performance, with their relative effectiveness varying across models and datasets, while remaining competitive with recent prompt optimization methods. The MoPrO-SD code and VaccineStance dataset are available on GitHub.

Meysam Sabbaghan, Nikesh Gyawali, Arman Zareian Jahromi +3

Kansas State University · University of Illinois Chicago

Stance Detection Prompt Optimization Large Language Model-Guided Optimization

Ask PDF

Jun 3, 2026

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set. PPI is provably unbiased regardless of the LLM judge's error profile. We make it applicable to hierarchical metrics like Precision@K, where annotations are per-document but the metric is per-query, by reducing the output-space computation from O(2^|C|) to O(2^K). On the ESCI benchmark, augmenting 30 human annotations with Claude 3 Sonnet judgments reduces the standard error of Precision@4 estimates from 4.45 to 3.50 (a 21% relative reduction). In a production system, our framework correctly identified the best of three system variants from 100 human labels and 2 hours of domain-expert annotation; A/B testing confirmed this ranking with +407 bps in daily sales.

Abhishek Divekar

Amazon

LLM-as-a-Judge IR Evaluation Prediction-Powered Inference

Ask PDF

Jun 11, 2026

SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection

Prompt-based LLMs are increasingly used for stance detection, but harder examples are not always repaired by clearer instructions, reasoning prompts, retrieval, or debate. We introduce SICI (Stance Inference Complexity Index), a seven-dimensional diagnostic measure of the semantic-pragmatic burden imposed by a target--text pair. Across SemEval-2016 and VAST, SICI predicts LLM accuracy better than surface proxies and shows substantial cross-scorer reliability (

α=0.771

). More importantly, LLM errors change regime as SICI increases: low-complexity examples invite over-attribution, especially Against predictions; intermediate examples form an unstable boundary; and high-complexity examples rapidly concentrate on None. This phase-transition-like structure persists across GPT-3.5, GPT-4o-mini, DeepSeek-V3, and GPT-4o, although stronger models move the boundaries. A 15-method intervention study further shows that prompting, retrieval, and debate often shift models along the attribution--abstention axis rather than removing the high-complexity bottleneck.

Fuqiang Niu, Bowen Zhang

School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China · School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China

LLM Evaluation Stance Detection

Ask PDF

Abstract

Language Model Error Detection

Explore similar work

MoPrO-SD: Modular Prompt Optimization for Stance Detection

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection