Organizations: Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, Odisha 751024, India. · IMC University of Applied Sciences Krems, Piaristengasse 1, 3500 Austria.
From customer feedback to social media, understanding human sentiment in text is central to how machines can interact meaningfully with people. However, despite notable progress, accurately capturing sentiment remains a challenging task, which continues to motivate further research in this area. To this end, we introduce Non-Differential Transformer (NDT). It is inspired by (but in contrast to) the state-of-the-art Differential Transformer (DT) model. While standard Transformers can struggle with irrelevant context, the sota DT model uses attention map subtraction, potentially for noise cancellation. We explore an alternative motivation, hypothesizing that benefits may arise from enabling different attention components to specialize on distinct concepts within the text, similar to multiplexing information channels or mixture models, rather than primarily canceling noise via subtraction. Guided by this concept-multiplexing (ConPlex) view, the specific architecture presented in this paper employs a purely additive strategy. It uses only positive weights, learned during training, to ensure constructive combination of these specialized attention perspectives. This design choice explores positive only integration, though our broader framework also shows promise with less constrained linear combinations involving both positive and negative weights. Our model computes attention via this positively weighted sum of multiple distinct attention maps. This allows the model to constructively integrate diverse signals and potentially capture more complex contextual relationships. Competitive performance is achieved by the proposed model for Sentiment Analysis while tested on multiple datasets. We conclude by presenting our results, challenges and future research agenda in this important area of research.
Figures & tables
Ncomp
Acc (%)
Precision
F1 Score
AUC
Vanilla Transformer
–
83.99
0.844
0.839
0.927
Differential Transformer
2
84.20
0.851
0.841
0.935
NDT [0,1]
2
85.56
0.856
0.856
0.933
TABLE I : Accuracy and Classification Metrics on IMDB Dataset
Ncomp
Layer 1 λ Values
Layer 2 λ Values
NDT [0,1]
2
(0.1507)
(0.1860)
3
(0.1506, 0.1436)
(0.1841, 0.1604)
4
(0.1515, 0.1436, 0.1365)
(0.1845, 0.1603, 0.1513)
NDT [-1,1]
2
(-0.2001)
(-0.0575)
TABLE II : Learned Lambda Values for the IMDB Dataset
Ncomp
Acc (%)
Precision
F1 Score
AUC
Vanilla Transformer
–
90.85
0.909
0.909
0.959
Differential Transformer
2
91.53
0.915
0.915
0.971
NDT [0,1]
2
91.74
0.917
0.917
0.964
TABLE III : Accuracy and Classification Metrics on SST-2 Dataset
Ncomp
Layer 1 λ Values
Layer 2 λ Values
NDT [0,1]
2
(0.1554)
(0.2032)
3
(0.1518, 0.1498)
(0.1990, 0.1746)
4
(0.1508, 0.1428, 0.1342)
(0.1987, 0.1721, 0.1591)
NDT [-1,1]
2
(-0.4230)
(0.0600)
TABLE IV : Learned Lambda Values for the SST-2 Dataset
Ncomp
Acc (%)
Precision
F1 Score
AUC
Vanilla Transformer
–
60.73
0.611
0.608
0.892
Differential Transformer
2
60.50
0.605
0.603
0.891
NDT [0,1]
2
60.36
0.603
0.603
0.891
TABLE V : Accuracy and Classification Metrics on YELP Reviews Dataset
Ncomp
Layer 1 λ Values
Layer 2 λ Values
NDT [0,1]
2
(0.1568)
(0.1926)
3
(0.1520, 0.1462)
(0.1907, 0.1635)
4
(0.1527, 0.1463, 0.1410)
(0.1894, 0.1634, 0.1558)
NDT [-1,1]
2
(-0.2003)
(-0.0468)
TABLE VI : Learned Lambda Values for the YELP Reviews Dataset
Ncomp
Acc (%)
Precision
F1 Score
AUC
Vanilla Transformer
–
81.99
0.817
0.817
0.936
Differential Transformer
2
82.33
0.819
0.821
0.945
NDT [0,1]
2
80.90
0.808
0.808
0.940
TABLE VII : Accuracy and Classification Metrics on Twitter Financial News Dataset
Ncomp
Layer 1 λ Values
Layer 2 λ Values
NDT [0,1]
2
(0.1522)
(0.1841)
3
(0.1512, 0.1444)
(0.1839, 0.1607)
4
(0.1516, 0.1441, 0.1372)
(0.1822, 0.1591, 0.1515)
NDT [-1,1]
2
(-0.2053)
(-0.0593)
TABLE VIII : Learned Lambda Values for the Twitter Financial News Dataset
Ncomp
IMDB Reviews
SST-2
YELP Reviews
Twitter Financial News
Vanilla Transformer
–
0.314
0.360
0.059
0.205
Differential Transformer
2
0.361
0.490
0.053
0.177
NDT [0,1]
2
0.318
0.467
0.055
0.114
TABLE IX : Silhouette Scores across Datasets
Fig. 1 : Confusion matrices for the best performing NDT model for each dataset. These visualizations illustrate how the model predictions are distributed across the ground truth classes.
Fig. 2 : ROC curves comparing the baseline DT models (left column) with the best performing NDT models (right column) for each dataset. Each pair of columns corresponds to a dataset, illustrating the relative discriminative power of NDT and DT across settings.
Fig. 3 : t-SNE visualizations of the learned embeddings for the best performing NDT model on each dataset. Distinct and compact clusters indicate effective class separation in the embedding space.
Transformers have dominated empirical machine learning models of natural language processing. In this paper, we introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications. Given that Transformers and related deep learning techniques might be evolving in ways we have never seen, we cannot dive into all the model details or cover all the technical areas. Instead, we focus on just those concepts that are helpful for gaining a good understanding of Transformers and their variants. We also summarize the key ideas that impact this field, thereby yielding some insights into the strengths and limitations of these models.
Tong Xiao, Jingbo Zhu
NLP Lab., Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China
Initially developed for natural language processing, Transformer architectures and attention mechanisms are now central to a wide range of deep learning models, including applications in time series forecasting. A standard attention mechanism, however, implicitly assumes homophilic interactions, limiting its ability to model data with positive and negative dependencies, such as time series. In this work, we introduce the Signed Dual Attention, a novel attention formulation that captures both positive and negative relational patterns without additional parameters. By leveraging a dual message-passing scheme inspired by correlation structures, Signed Dual Attention propagates both supportive and contrastive information within a single shared block, effectively achieving the expressiveness of two head attention without additional parameters. This module can be seamlessly integrated into existing architectures and can yield performance gains in certain situations, requiring signed relational modeling. This approach opens a pathway toward more expressive and parameter-efficient transformers.
Balthazar Courvoisier, Tristan Cazenave
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France
Aspect-Term Sentiment Analysis (ATSA) in multi-aspect sentences faces a fundamental tradeoff between efficiency and expressiveness. Existing models either re-encode the sentence for each aspect or rely on static use of deep representations, leading to redundant computation and limited adaptivity. We argue that Transformer depth is a costly, queryable resource, and propose DABS, a single-pass inference framework that encodes each sentence once to construct a reusable, depth-ordered substrate. Each aspect then queries this shared representation to selectively read relevant tokens and abstraction levels, without re-encoding. This decouples shared sentence encoding from lightweight, aspect-conditioned readout. Experiments on four ATSA benchmarks show that DABS achieves competitive performance while reducing end-to-end computation by up to 60% in multi-aspect settings (M >= 2). Further analyses indicate that adaptive depth querying is most beneficial for linguistically complex cases such as negation and contrast. Code is publicly available at https://github.com/panzhzh/acl-dabs
Yan Xia, Zhuangzhuang Pan, Amirrudin Kamsin +1
1Universiti Malaya, Malaysia · 2Suzhou University of Technology, China · 3VinUniversity, Vietnam