LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines

Jiechao Gao; Rohan Kumar Yadav; Yuangang Li; Yuandong Pan; Jie Wang; Ying Liu; Michael Lepech

cs.CLApr 14, 2026

LLM-Guided Semantic Bootstrapping for Interpretable Text Classification with Tsetlin Machines

Authors: Jiechao Gao, Rohan Kumar Yadav, Yuangang Li, Yuandong Pan, Jie Wang, Ying Liu, Michael Lepech

Organizations: Stanford University · Independent Researcher · University of California, Irvine · University of the Chinese Academy of Sciences

Abstract

Pretrained language models (PLMs) like BERT provide strong semantic representations but are costly and opaque, while symbolic models such as the Tsetlin Machine (TM) offer transparency but lack semantic generalization. We propose a semantic bootstrapping framework that transfers LLM knowledge into symbolic form, combining interpretability with semantic capacity. Given a class label, an LLM generates sub-intents that guide synthetic data creation through a three-stage curriculum (seed, core, enriched), expanding semantic diversity. A Non-Negated TM (NTM) learns from these examples to extract high-confidence literals as interpretable semantic cues. Injecting these cues into real data enables a TM to align clause logic with LLM-inferred semantics. Our method requires no embeddings or runtime LLM calls, yet equips symbolic models with pretrained semantic priors. Across multiple text classification tasks, it improves interpretability and accuracy over vanilla TM, achieving performance comparable to BERT while remaining fully symbolic and efficient.

Figures & tables

Input	Type I Feedback			Type II Feedback
	Clause	1	0	Clause	1	0
	Literal	1 0	1 0	Literal	1 0	1 0
Include Literal	P(Reward)	$\tfrac{s-1}{s}$ NA	0 0	P(Reward)	0 NA	0 0
	P(Inaction)	$\tfrac{1}{s}$ NA	$\tfrac{s-1}{s}$ $\tfrac{s-1}{s}$	P(Inaction)	1.0 NA	1.0 1.0
	P(Penalty)	0 NA	$\tfrac{1}{s}$ $\tfrac{1}{s}$	P(Penalty)	0 NA	0 0
Exclude Literal	P(Reward)	0 $\tfrac{1}{s}$	$\tfrac{1}{s}$ $\tfrac{1}{s}$	P(Reward)	0 0	0 0

Table 1: The Type I and Type II Feedback.

Model/Datasets	AG-News	R8	R52	IMDB	SST2	HoC
BoW TFIDF	89.64	93.74	86.95	85.30	75.50	73.4
char-CNN	87.20	94.02	85.37	80.35	80.78	76.60
fastText, h = 10	91.50	98.10	94.74	–	80.1	–
RNN/LSTM	86.06	93.68	85.54	78.82	77.50	75.70
RNN/LSTM (GloVe)	92.10	96.09	90.48	80.72	80.35	76.26
BERT	94.75	97.49	94.26	93.46	94.00	82.90

Table 2: Performance of the proposed LLM-Guided TM on selected classification datasets. The bold entries indicate the highest performance, while bold and underlined indicate the second highest. A dash (–) indicates missing values not reported in literature. LLM-Guided TM results are averaged over 5 runs, and the includes standard deviation. Baseline results are drawn from literature and do not report variances, so significance testing could not be applied.

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 226 papers

Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability
Jun 18, 2026Jiechao Gao, Rohan Kumar Yadav, Yuangang Li +4Interpretable ML Text Classification
Stanford University · Independent Researcher · University of California, Irvine +1
Structured Prompt Optimization Meets Reinforcement Learning for Global and Local Interpretability over Complex Text
May 27, 2026Tianyang Zhou, Wenbo Chen, Pierre Jinghong Liang +1Explainable Text Classification LLM Interpretability
1Carnegie Mellon University · 2Amazon
BoolXLLM: LLM-Assisted Explainability for Boolean Models
May 12, 2026Du Cheng, Serdar Kadioglu, Xin WangExplainable Artificial Intelligence Boolean Function Learning
AI Center of Excellence, Fidelity Investments, MA, USA · Department of Computer Science, Brown University, RI, USA

Jun 18, 2026J/K move · Enter open · S save

Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

Jiechao Gao, Rohan Kumar Yadav, Yuangang Li, Yuandong Pan+3

Stanford University · Independent Researcher · University of California, Irvine+1

Interpretable ML Text Classification Semantic Clustering Classification

Open paper AskPDF arXiv

Pre-trained language models such as BERT achieve strong text classification performance but lack transparency, limiting their use in high-stakes settings. The Tsetlin Machine (TM) offers fully interpretable, clause-based reasoning but captures little semantic information, and prior attempts to bridge the two rely on static word embeddings that miss contextual meaning. We propose a semantic pre-training framework that transfers knowledge from a pre-trained language model into a TM without using embeddings. Text samples are grouped into semantically coherent clusters with K-means or Top2Vec, and the resulting cluster-sample pairs pre-train a non-negated TM with enhanced Type I feedback. The TM thereby learns interpretable semantic keywords that are fine-tuned on downstream tasks. Across five datasets, our method substantially outperforms vanilla and embedding-based TMs and reaches performance competitive with BERT while remaining interpretable.

Abstract

Text Classification

Explore similar work

Clusters are All You Need: Pre-Training the Tsetlin Machine with Semantic Clusters from Language Models for Interpretability

Structured Prompt Optimization Meets Reinforcement Learning for Global and Local Interpretability over Complex Text

BoolXLLM: LLM-Assisted Explainability for Boolean Models