BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs

Ilias Aarab

cs.CLMar 12, 2026

BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs

Authors: Ilias Aarab

Organizations: European Central Bank, Frankfurt am Main, Germany

Abstract

Zero-shot text classification (ZSC) offers the promise of eliminating costly task-specific annotation by matching texts directly to human-readable label descriptions. While early approaches have predominantly relied on cross-encoder models fine-tuned for natural language inference (NLI), recent advances in text-embedding models, rerankers, and instruction-tuned large language models (LLMs) have challenged the dominance of NLI-based architectures. Yet, systematically comparing these diverse approaches remains difficult. Existing evaluations, such as MTEB, often incorporate labeled examples through supervised probes or fine-tuning, leaving genuine zero-shot capabilities underexplored. To address this, we introduce BTZSC, a comprehensive benchmark of 22 public datasets spanning sentiment, topic, intent, and emotion classification, capturing diverse domains, class cardinalities, and document lengths. Leveraging BTZSC, we conduct a systematic comparison across four major model families, NLI cross-encoders, embedding models, rerankers and instruction-tuned LLMs, encompassing 38 public and custom checkpoints. Our results show that: (i) modern rerankers, exemplified by Qwen3-Reranker-8B, set a new state-of-the-art with macro F1 = 0.72; (ii) strong embedding models such as GTE-large-en-v1.5 substantially close the accuracy gap while offering the best trade-off between accuracy and latency; (iii) instruction-tuned LLMs at 4--12B parameters achieve competitive performance (macro F1 up to 0.67), excelling particularly on topic classification but trailing specialized rerankers; (iv) NLI cross-encoders plateau even as backbone size increases; and (v) scaling primarily benefits rerankers and LLMs over embedding models. BTZSC and accompanying evaluation code are publicly released to support fair and reproducible progress in zero-shot text understanding.

Figures & tables

Emotion
Domain	Dataset	Num Classes	Avg Token Count
dialogue	empathetic	32	132
social-media	dair_ai_emotion	6	20
Intent
banking	banking77	77	13
social-media	biasframes_intent	2	27

Table 1: Summary statistics of BTZSC datasets.

Figure 1: Pairwise weighted Jaccard similarity between datasets.

Panel of Figure 2 — Figure 2: (a) Performance of NLI-based cross-encoders on BTZSC; points are individual datasets and diamonds mark task-wise medians, comparing model size (base vs. large) and loss type (binary vs. three-way). (b) Effect of scale on zero-shot performance: macro- $F_{1}$ vs. parameter count (log scale); bands show 95% CIs.

Panel of Figure 3 — Figure 3: (a) Performance–speed trade-off: macro-F 1 on BTZSC vs. normalized inference throughput ( $1/$ wall time) on a standard test set; the upper-right quadrant (split by metric medians) marks models with the best accuracy–efficiency balance. (b) NLI ability vs. zero-shot classification: AUROC on standard NLI benchmarks (x-axis) vs. BTZSC macro-F 1 (y-axis).

Base encoders
Model	Topic	Sentiment	Intent	Emotion	Avg F1	Avg Acc
bert-large-uncased	0.34 (0.22)	0.38 (0.06)	0.15 (0.24)	0.08 (0.11)	0.30 (0.20)	0.40 (0.26)
deberta-v3-large	0.30 (0.23)	0.34 (0.03)	0.16 (0.26)	0.05 (0.07)	0.27 (0.20)	0.36 (0.25)
ModernBERT-large	0.30 (0.24)	0.37 (0.06)	0.14 (0.21)	0.03 (0.04)	0.27 (0.21)	0.35 (0.24)
NLI cross-encoders
bart-large-mnli	0.37 (0.23)	0.84 (0.19)	0.43 (0.16)	0.41 (0.04)	0.51 (0.28)	0.53 (0.28)

Table 2: Zero-shot classification results on BTZSC. We report macro F1 per task family, overall macro F1 (Avg F1), and micro accuracy (Avg Acc). Parentheses report the across-dataset standard deviation within each task family reflecting task heterogeneity. Bold indicates the best and underlining the second-best score per column; the best model within each family (based on overall macro F1) is also underlined.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Dataset	Source	License
Emotion
dialogue	empathetic_dialogues	( Rashkin et al., 2019 )	CC BY-NC 4.0
social-media	dair_ai_emotion	( Saravia et al., 2018 )	Research/education only
Intent
banking	banking77	( Casanueva et al., 2020 )	CC BY 4.0
social-media	biasframes_intent	( Sap et al., 2020 )	CC BY 4.0

Appendix

Table 3: Sources and licenses for BTZSC datasets. All datasets are used as single-label classification tasks and evaluated in a purely zero-shot setting on their respective test splits (Section A.1 ).

Model	Yr	Arch.	Backbone	FT / train data	# P	Pool / dim
Base encoders
bert-large-uncased	2018	enc.	BERT	NA	340M	-
deberta-v3-large	2021	enc.	DeBERTa v3	NA	304M	-
ModernBERT-large	2024	enc.	ModernBERT	NA	395M	-
NLI cross-encoders
bart-large-mnli	2020	enc-dec.	BART	SNLI, MNLI	406M	-

Appendix

Table 4: Architectural and training overview of the 38 models evaluated. Columns list publication year (Yr), encoder/decoder architecture (Arch.), backbone, principal fine-tuning data, parameter count (#P), and pooling strategy with embedding dimensionality.

Base encoders
Task	Topic											Sentiment						Intent			Emotion
Model	AGN	BF-Off	BF-Sex	CAPS	MAN	TT	WT-Ins	WT-Obs	WT-Thr	WT-Agg	YT	AmzPol	AppR	FPB	IMDB	RT	Yelp	B77	BF-Int	MASS	EmoD	Emp	Avg F1
bert-large-uncased	0.17	0.48	0.48	0.03	0.01	0.33	0.51	0.58	0.49	0.53	0.08	0.42	0.36	0.28	0.38	0.47	0.36	0.02	0.42	0.00	0.16	0.00	0.30
deberta-v3-large	0.18	0.49	0.12	0.01	0.01	0.34	0.55	0.56	0.51	0.53	0.03	0.33	0.37	0.30	0.33	0.35	0.37	0.01	0.45	0.00	0.10	0.00	0.27
ModernBERT-large	0.10	0.53	0.46	0.01	0.01	0.36	0.46	0.58	0.13	0.60	0.08	0.33	0.44	0.30	0.34	0.36	0.43	0.02	0.38	0.02	0.06	0.01	0.27
NLI cross-encoders

Appendix

Table 5: Zero-shot classification results on BTZSC by dataset (macro-averaged F1). Abbreviations: AGN = AGNews , BF-Off / BF-Sex = BiasFrames (offensive / sex), CAPS = CAPSOTU , MAN = Manifesto, TT = TrueTeacher, WT-Ins / WT-Obs / WT-Thr / WT-Agg = WikiToxic (insult / obscene / threat / toxic aggregated), YT = Yahoo Topics, AmzPol = Amazon Polarity, AppR = AppReviews, FPB = FinancialPhraseBank, RT = RottenTomatoes, Yelp = YelpReviews, B77 = Banking77, BF-Int = BiasFrames (intent), MASS = MASSIVE (intent), EmoD = EmotionDiary, Emp = EmpatheticDialogues.

Base encoders
Task	Topic / toxicity											Sentiment						Intent			Emotion
Model	AGN	BF-Off	BF-Sex	CAPS	MAN	TT	WT-Ins	WT-Obs	WT-Thr	WT-Agg	YT	AmzPol	AppR	FPB	IMDB	RT	Yelp	B77	BF-Int	MASS	EmoD	Emp	Avg Acc
bert-large-uncased	0.26	0.57	0.91	0.09	0.02	0.49	0.58	0.58	0.78	0.55	0.13	0.48	0.51	0.56	0.52	0.52	0.52	0.03	0.44	0.01	0.20	0.03	0.40
deberta-v3-large	0.28	0.58	0.12	0.01	0.04	0.51	0.65	0.64	0.86	0.58	0.11	0.49	0.51	0.40	0.49	0.50	0.52	0.03	0.47	0.00	0.20	0.03	0.36
ModernBERT-large	0.24	0.53	0.72	0.03	0.01	0.51	0.46	0.59	0.14	0.60	0.15	0.49	0.50	0.57	0.51	0.50	0.51	0.03	0.48	0.02	0.14	0.02	0.35
NLI cross-encoders

Appendix

Table 6: Zero-shot classification results on BTZSC by dataset (micro-averaged accuracy). Abbreviations: AGN = AGNews , BF-Off / BF-Sex = BiasFrames (offensive / sex), CAPS = CAPSOTU , MAN = Manifesto, TT = TrueTeacher, WT-Ins / WT-Obs / WT-Thr / WT-Agg = WikiToxic (insult / obscene / threat / toxic aggregated), YT = Yahoo Topics, AmzPol = Amazon Polarity, AppR = AppReviews, FPB = FinancialPhraseBank, RT = RottenTomatoes, Yelp = YelpReviews, B77 = Banking77, BF-Int = BiasFrames (intent), MASS = MASSIVE (intent), EmoD = EmotionDiary, Emp = EmpatheticDialogues.

Base encoders
Task	Topic / toxicity											Sentiment						Intent			Emotion
Model	AGN	BF-Off	BF-Sex	CAPS	MAN	TT	WT-Ins	WT-Obs	WT-Thr	WT-Agg	YT	AmzPol	AppR	FPB	IMDB	RT	Yelp	B77	BF-Int	MASS	EmoD	Emp	Avg R
bert-large-uncased	0.26	0.53	0.49	0.10	0.02	0.50	0.52	0.58	0.55	0.55	0.12	0.47	0.51	0.33	0.51	0.52	0.51	0.03	0.46	0.03	0.18	0.03	0.35
deberta-v3-large	0.27	0.53	0.50	0.04	0.02	0.50	0.58	0.58	0.52	0.57	0.09	0.50	0.51	0.39	0.50	0.50	0.51	0.03	0.49	0.00	0.18	0.03	0.36
ModernBERT-large	0.25	0.55	0.50	0.07	0.01	0.50	0.47	0.59	0.49	0.61	0.14	0.50	0.50	0.33	0.50	0.50	0.52	0.04	0.52	0.03	0.18	0.02	0.36
NLI cross-encoders

Appendix

Table 7: Zero-shot classification results on BTZSC by dataset (macro-averaged recall). Abbreviations: AGN = AGNews , BF-Off / BF-Sex = BiasFrames (offensive / sex), CAPS = CAPSOTU , MAN = Manifesto, TT = TrueTeacher, WT-Ins / WT-Obs / WT-Thr / WT-Agg = WikiToxic (insult / obscene / threat / toxic aggregated), YT = Yahoo Topics, AmzPol = Amazon Polarity, AppR = AppReviews, FPB = FinancialPhraseBank, RT = RottenTomatoes, Yelp = YelpReviews, B77 = Banking77, BF-Int = BiasFrames (intent), MASS = MASSIVE (intent), EmoD = EmotionDiary, Emp = EmpatheticDialogues.

Base encoders
Task	Topic / toxicity											Sentiment						Intent			Emotion
Model	AGN	BF-Off	BF-Sex	CAPS	MAN	TT	WT-Ins	WT-Obs	WT-Thr	WT-Agg	YT	AmzPol	AppR	FPB	IMDB	RT	Yelp	B77	BF-Int	MASS	EmoD	Emp	Avg P
bert-large-uncased	0.18	0.55	0.47	0.04	0.03	0.25	0.53	0.58	0.51	0.56	0.11	0.45	0.67	0.38	0.55	0.53	0.58	0.02	0.45	0.00	0.19	0.02	0.35
deberta-v3-large	0.38	0.56	0.50	0.01	0.02	0.25	0.66	0.66	0.51	0.62	0.04	0.24	0.58	0.32	0.24	0.53	0.58	0.02	0.49	0.00	0.10	0.00	0.33
ModernBERT-large	0.06	0.55	0.50	0.02	0.02	0.53	0.47	0.58	0.50	0.61	0.07	0.24	0.49	0.29	0.26	0.50	0.56	0.05	0.59	0.04	0.07	0.01	0.32
NLI cross-encoders

Appendix

Table 8: Zero-shot classification results on BTZSC by dataset (macro-averaged precision). Abbreviations: AGN = AGNews , BF-Off / BF-Sex = BiasFrames (offensive / sex), CAPS = CAPSOTU , MAN = Manifesto, TT = TrueTeacher, WT-Ins / WT-Obs / WT-Thr / WT-Agg = WikiToxic (insult / obscene / threat / toxic aggregated), YT = Yahoo Topics, AmzPol = Amazon Polarity, AppR = AppReviews, FPB = FinancialPhraseBank, RT = RottenTomatoes, Yelp = YelpReviews, B77 = Banking77, BF-Int = BiasFrames (intent), MASS = MASSIVE (intent), EmoD = EmotionDiary, Emp = EmpatheticDialogues.

Model	AmzCf	MASS-S	MTOP-D	ToxicConv	IMDB	TweetSentExt	B77	MASS-I	Avg F1
Base encoders
bert-large-uncased	0.25	0.03	0.08	0.44	0.38	0.41	0.02	0.00	0.20
deberta-v3-large	0.51	0.02	0.07	0.49	0.33	0.31	0.01	0.00	0.22
ModernBERT-large	0.45	0.07	0.11	0.43	0.34	0.16	0.02	0.02	0.20
NLI cross-encoders
bart-large-mnli	0.15	0.54	0.84	0.50	0.93	0.54	0.28	0.41	0.52

Appendix

Table 9: Zero-shot classification results (macro-averaged F1) on eight English classification datasets from MTEB v2 plus their average (Avg F1). Bold denotes the best and underlining the second-best score in each column. Best model in each family is underlined.

Base encoders
Model	AmzCf	MASS-S	MTOP-D	ToxicConv	IMDB	TweetSE	B77	MASS-I	Avg Acc
bert-large-uncased	0.25	0.05	0.15	0.53	0.52	0.41	0.03	0.01	0.25
deberta-v3-large	0.66	0.03	0.09	0.50	0.49	0.32	0.03	0.00	0.27
ModernBERT-large	0.83	0.09	0.13	0.50	0.51	0.33	0.03	0.02	0.30
NLI cross-encoders
bart-large-mnli	0.18	0.52	0.84	0.57	0.93	0.62	0.29	0.43	0.55

Appendix

Table 10: Zero-shot classification results (micro accuracy) on the 8 English MTEB-v2 classification tasks. Bold denotes the best and underlining the second-best score in each column. Best model in each family is underlined.

Base encoders
Model	AmzCf	MASS-S	MTOP-D	ToxicConv	IMDB	TweetSE	B77	MASS-I	Avg Acc
bert-large-uncased	0.49	0.06	0.11	0.53	0.51	0.42	0.03	0.03	0.27
deberta-v3-large	0.52	0.05	0.08	0.50	0.50	0.32	0.03	0.00	0.25
ModernBERT-large	0.50	0.10	0.15	0.50	0.50	0.33	0.04	0.03	0.27
NLI cross-encoders
bart-large-mnli	0.50	0.55	0.84	0.57	0.93	0.62	0.30	0.43	0.59

Appendix

Table 11: Zero-shot classification results (macro-averaged recall) on the 8 MTEB (English, v2) classification datasets and their average (Avg Recall). Bold denotes the best and underlining the second-best score in each column. Best model in each family is underlined.

Model	Amazon C.	Massive S.	MTOP	Toxic	IMDB	Tweet S.	Banking77	Massive I.	Avg Prec
Base encoders
bert-large-uncased	0.49	0.04	0.11	0.60	0.55	0.46	0.02	0.00	0.28
deberta-v3-large	0.51	0.08	0.07	0.51	0.24	0.32	0.02	0.00	0.22
ModernBERT-large	0.41	0.08	0.22	0.49	0.26	0.11	0.05	0.04	0.21
NLI cross-encoders
bart-large-mnli	0.59	0.67	0.84	0.68	0.93	0.64	0.47	0.56	0.67

Appendix

Table 12: Zero-shot classification results (macro-averaged precision) on BTZSC for the eight MTEB (EN, v2) classification datasets. We report per-dataset precision and overall average precision (Avg Prec). Bold denotes the best and underlining the second-best score in each column. Best model in each family is underlined.

Zero-Shot Text Classification

Zero-shot text classification assigns texts to categories without task-specific labeled training data, using language models, embeddings, or label descriptions.

See all 34 papers

Oct 12, 2026

Label-Centric Knowledge Retrieval for Zero-Shot Text Classification

Flagship large language models (LLMs) offer rich general knowledge and strong reasoning capabilities, but their high inference cost and latency limit their use for large-scale data labeling. We study zero-shot text classification with semantically meaningful labels and task descriptions, without task-specific labeled examples or classifier feedback. We propose SLKR (Semantic Label Knowledge Retrieval), a label-centric retrieval-augmented generation framework that separates offline knowledge construction from lightweight online inference. Offline, a flagship LLM independently generates label definitions, retrieval keywords, and boundary rules for potentially confusable label pairs, forming a reusable label-knowledge store. Because this construction operates over the label space rather than the input corpus, its one-time flagship cost can be amortized across large-scale inference. Online, hybrid semantic and lexical retrieval first identifies the top-

K

candidate labels and then selects up to

M

input-relevant boundary rules connecting candidate-label pairs. A lightweight LLM performs the final classification in a single call using the retrieved definitions and rules. Across three public benchmarks and a real-world multilingual book-classification dataset, SLKR achieves the best results on all four datasets with Qwen3.5-4B and on three of four with Ministral-3-3B, while using compact input contexts and no task-specific labeled examples. Boundary ablations further show that retrieving a small set of input-relevant rules preserves performance close to using all boundaries with substantially less context, supporting efficient reuse of flagship-generated label knowledge for lightweight classification.

Daoping Wu, Dong Wang, Siyu Liu +5

Zhejiang Lab, Hangzhou, China · School of Computer Science and Software Engineering, Harbin Institute of Technology, China · School of Mechanical Engineering, Shanghai Jiao Tong University, China

Zero-Shot Text Classification Retrieval-Augmented Classification Efficient Language Model Inference

Ask PDF

Apr 6, 2026

Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

Large language models encode extensive world knowledge valuable for zero-shot named entity recognition. However, their causal attention mechanism, where tokens attend only to preceding context, prevents effective token classification when disambiguation requires future context. Existing approaches use LLMs generatively, prompting them to list entities or produce structured outputs, but suffer from slow autoregressive decoding, hallucinated entities, and formatting errors. We propose Just Pass Twice (JPT), a simple yet effective method that enables causal LLMs to perform discriminative token classification with full bidirectional context. Our key insight is that concatenating the input to itself lets each token in the second pass attend to the complete sentence, requiring no architectural modifications. We combine these representations with definition-guided entity embeddings for flexible zero-shot generalization. Our approach achieves state-of-the-art results on zero-shot NER benchmarks, surpassing the previous best method by +7.9 F1 on average across CrossNER and MIT benchmarks, being over 20x faster than comparable generative methods. Interactive demo and access to the model are available at https://witness-ai-jpt-ner.hf.space/

Ahmed Ewais, Ahmed Hashish, Amr Ali

WitnessAI

Named Entity Recognition Decoder-Only Language Models Efficient Language Model Inference

Ask PDF

Sep 23, 2026

Zero-Shot Visualization: Exploring Text Corpora with User-Prompted Axes

We study the application of large language models (LLMs) to the visual exploration of textual corpora. We introduce zero-shot visualization (ZSV), a task in which users specify concepts in natural language and documents are mapped onto the corresponding concept axes for visualization. Building a ZSV system of practical value is non-trivial, as it requires choices at the intersection of feature functions, efficient implementation tradeoffs, and pre/post-processing decisions affecting visualization quality. To that end, we establish a benchmark that compares methods spanning embedding similarity, direct semantic judgments, and conditional likelihood estimation in this setting. Across multiple datasets and use cases we evaluate the properties of different scoring methods and design choices in terms of semantic faithfulness, score fidelity, and computational cost. Our results identify that scoring based on next-token probabilities offers the strongest practical trade-off among the evaluated methods. We further apply this approach to unlabeled corpora to examine its behavior in realistic exploratory settings. These experiments highlight additional design considerations, including the use of graded axes together with binary relevance filtering, and reveal a compositional sentiment bias in off-topic documents. Based on these findings, we provide practical guidelines for constructing end-to-end ZSV baselines.

Arnau Bueno Tricas, Jose A. Rodríguez-Serrano

Universitat Ramon Llull Esade Barcelona, Spain

Topic pending

Ask PDF

Abstract

Zero-Shot Text Classification

Explore similar work

Label-Centric Knowledge Retrieval for Zero-Shot Text Classification

Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER

Zero-Shot Visualization: Exploring Text Corpora with User-Prompted Axes