NDT: Non-Differential Transformer and Its Application to Sentiment Analysis

Soudeep Ghoshal; Himanshu Buckchash; Sarita Paudel; Rubén Ruiz-Torrubiano

cs.IRMar 21, 2026

NDT: Non-Differential Transformer and Its Application to Sentiment Analysis

Authors: Soudeep Ghoshal, Himanshu Buckchash, Sarita Paudel, Rubén Ruiz-Torrubiano

Organizations: Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, Odisha 751024, India. · IMC University of Applied Sciences Krems, Piaristengasse 1, 3500 Austria.

Abstract

From customer feedback to social media, understanding human sentiment in text is central to how machines can interact meaningfully with people. However, despite notable progress, accurately capturing sentiment remains a challenging task, which continues to motivate further research in this area. To this end, we introduce Non-Differential Transformer (NDT). It is inspired by (but in contrast to) the state-of-the-art Differential Transformer (DT) model. While standard Transformers can struggle with irrelevant context, the sota DT model uses attention map subtraction, potentially for noise cancellation. We explore an alternative motivation, hypothesizing that benefits may arise from enabling different attention components to specialize on distinct concepts within the text, similar to multiplexing information channels or mixture models, rather than primarily canceling noise via subtraction. Guided by this concept-multiplexing (ConPlex) view, the specific architecture presented in this paper employs a purely additive strategy. It uses only positive weights, learned during training, to ensure constructive combination of these specialized attention perspectives. This design choice explores positive only integration, though our broader framework also shows promise with less constrained linear combinations involving both positive and negative weights. Our model computes attention via this positively weighted sum of multiple distinct attention maps. This allows the model to constructively integrate diverse signals and potentially capture more complex contextual relationships. Competitive performance is achieved by the proposed model for Sentiment Analysis while tested on multiple datasets. We conclude by presenting our results, challenges and future research agenda in this important area of research.

Figures & tables

$N_{comp}$	Acc (%)	Precision	F1 Score	AUC
Vanilla Transformer
–	83.99	0.844	0.839	0.927
Differential Transformer
2	84.20	0.851	0.841	0.935
NDT [0,1]
2	85.56	0.856	0.856	0.933

TABLE I : Accuracy and Classification Metrics on IMDB Dataset

$N_{comp}$	Layer 1 $\lambda$ Values	Layer 2 $\lambda$ Values
NDT [0,1]
2	(0.1507)	(0.1860)
3	(0.1506, 0.1436)	(0.1841, 0.1604)
4	(0.1515, 0.1436, 0.1365)	(0.1845, 0.1603, 0.1513)
NDT [-1,1]
2	(-0.2001)	(-0.0575)

TABLE II : Learned Lambda Values for the IMDB Dataset

$N_{comp}$	Acc (%)	Precision	F1 Score	AUC
Vanilla Transformer
–	90.85	0.909	0.909	0.959
Differential Transformer
2	91.53	0.915	0.915	0.971
NDT [0,1]
2	91.74	0.917	0.917	0.964

TABLE III : Accuracy and Classification Metrics on SST-2 Dataset

$N_{comp}$	Layer 1 $\lambda$ Values	Layer 2 $\lambda$ Values
NDT [0,1]
2	(0.1554)	(0.2032)
3	(0.1518, 0.1498)	(0.1990, 0.1746)
4	(0.1508, 0.1428, 0.1342)	(0.1987, 0.1721, 0.1591)
NDT [-1,1]
2	(-0.4230)	(0.0600)

TABLE IV : Learned Lambda Values for the SST-2 Dataset

$N_{comp}$	Acc (%)	Precision	F1 Score	AUC
Vanilla Transformer
–	60.73	0.611	0.608	0.892
Differential Transformer
2	60.50	0.605	0.603	0.891
NDT [0,1]
2	60.36	0.603	0.603	0.891

TABLE V : Accuracy and Classification Metrics on YELP Reviews Dataset

$N_{comp}$	Layer 1 $\lambda$ Values	Layer 2 $\lambda$ Values
NDT [0,1]
2	(0.1568)	(0.1926)
3	(0.1520, 0.1462)	(0.1907, 0.1635)
4	(0.1527, 0.1463, 0.1410)	(0.1894, 0.1634, 0.1558)
NDT [-1,1]
2	(-0.2003)	(-0.0468)

TABLE VI : Learned Lambda Values for the YELP Reviews Dataset

$N_{comp}$	Acc (%)	Precision	F1 Score	AUC
Vanilla Transformer
–	81.99	0.817	0.817	0.936
Differential Transformer
2	82.33	0.819	0.821	0.945
NDT [0,1]
2	80.90	0.808	0.808	0.940

TABLE VII : Accuracy and Classification Metrics on Twitter Financial News Dataset

$N_{comp}$	Layer 1 $\lambda$ Values	Layer 2 $\lambda$ Values
NDT [0,1]
2	(0.1522)	(0.1841)
3	(0.1512, 0.1444)	(0.1839, 0.1607)
4	(0.1516, 0.1441, 0.1372)	(0.1822, 0.1591, 0.1515)
NDT [-1,1]
2	(-0.2053)	(-0.0593)

TABLE VIII : Learned Lambda Values for the Twitter Financial News Dataset

$N_{comp}$	IMDB Reviews	SST-2	YELP Reviews	Twitter Financial News
Vanilla Transformer
–	0.314	0.360	0.059	0.205
Differential Transformer
2	0.361	0.490	0.053	0.177
NDT [0,1]
2	0.318	0.467	0.055	0.114

TABLE IX : Silhouette Scores across Datasets

Panel of Figure 10 — Fig. 1 : Confusion matrices for the best performing NDT model for each dataset. These visualizations illustrate how the model predictions are distributed across the ground truth classes.

Panel of Figure 11 — Fig. 2 : ROC curves comparing the baseline DT models (left column) with the best performing NDT models (right column) for each dataset. Each pair of columns corresponds to a dataset, illustrating the relative discriminative power of NDT and DT across settings.

Panel of Figure 12 — Fig. 3 : t-SNE visualizations of the learned embeddings for the best performing NDT model on each dataset. Distinct and compact clusters indicate effective class separation in the embedding space.

Model ( $N_{comp}$ )	IMDB Params (M)	SST-2 Params (M)	YELP Params (M)	Twitter Financial News Params (M)	$\Delta$ Params (M)
Vanilla (–)	3.63	1.40	8.38	0.67	–
DT (2)	3.79	1.56	8.54	0.82	$\sim 0.16$
NDT (2)	3.79	1.56	8.54	0.82	$\sim 0.16$
NDT (3)	3.83	1.60	8.58	0.85	$\sim 0.20$
NDT (4)	3.86	1.63	8.61	0.88	$\sim 0.23$

TABLE X : Parameter Overhead Analysis

Model ( $N_{comp}$ )	IMDB Time (ms)	SST-2 Time (ms)	YELP Time (ms)	Twitter Financial News Time (ms)
Vanilla (–)	25.58	$\sim$ 6.25	75.42	$\sim$ 0
DT (2)	38.36	$\sim$ 6.25	121.65	$\sim$ 0
NDT (2)	40.92	$\sim$ 6.25	126.52	$\sim$ 0
NDT (3)	48.59	$\sim$ 6.25	158.15	$\sim$ 0
NDT (4)	58.82	$\sim$ 6.25	197.08	$\sim$ 0

TABLE XI : Inference Time Comparison per Batch

Explore similar work

Introduction to Transformers: an NLP Perspective
Nov 29, 2023Tong Xiao, Jingbo ZhuTransformer Transformer Attention
NLP Lab., Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China
Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting
Jun 3, 2026Balthazar Courvoisier, Tristan CazenaveTime Series Forecasting Transformer Attention
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France
Single-Pass, Depth-Selective Reading for Multi-Aspect Sentiment Analysis
May 20, 2026Yan Xia, Zhuangzhuang Pan, Amirrudin Kamsin +1Aspect-Based Sentiment Analysis Efficient Language Model Inference
1Universiti Malaya, Malaysia · 2Suzhou University of Technology, China · 3VinUniversity, Vietnam

Nov 29, 2023J/K move · Enter open · S save

Introduction to Transformers: an NLP Perspective

Tong Xiao, Jingbo Zhu

NLP Lab., Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

Transformer Transformer Attention

Open paper AskPDF arXiv

Transformers have dominated empirical machine learning models of natural language processing. In this paper, we introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications. Given that Transformers and related deep learning techniques might be evolving in ways we have never seen, we cannot dive into all the model details or cover all the technical areas. Instead, we focus on just those concepts that are helpful for gaining a good understanding of Transformers and their variants. We also summarize the key ideas that impact this field, thereby yielding some insights into the strengths and limitations of these models.

Abstract

Explore similar work

Introduction to Transformers: an NLP Perspective

Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting

Single-Pass, Depth-Selective Reading for Multi-Aspect Sentiment Analysis