Explicit Grammar Semantic Feature Fusion for Robust Text Classification

Azrin Sultana; Firoz Ahmed

cs.CLFeb 24, 2026

Explicit Grammar Semantic Feature Fusion for Robust Text Classification

Authors: Azrin Sultana, Firoz Ahmed

Organizations: Department of Computer Science, American International University-Bangladesh, Dhaka-1229, Bangladesh

Abstract

Natural Language Processing enables computers to understand human language by analysing and classifying text efficiently with deep-level grammatical and semantic features. Existing models capture features by learning from large corpora with transformer models, which are computationally intensive and unsuitable for resource-constrained environments. Therefore, our proposed study incorporates comprehensive grammatical rules alongside semantic information to build a robust, lightweight classification model without resorting to full parameterised transformer models or heavy deep learning architectures. The novelty of our approach lies in its explicit encoding of sentence-level grammatical structure, including syntactic composition, phrase patterns, and complexity indicators, into a compact grammar vector, which is then fused with frozen contextual embeddings. These heterogeneous elements unified a single representation that captures both the structural and semantic characteristics of the text. Deep learning models such as Deep Belief Networks (DBNs), Long Short-Term Memory (LSTMs), BiLSTMs, and transformer-based BERT and XLNET were used to train and evaluate the model, with the number of epochs varied. Based on experimental results, the unified feature representation model captures both the semantic and structural properties of text, outperforming baseline models by 2%-15%, enabling more effective learning across heterogeneous domains. Unlike prior syntax-aware transformer models that inject grammatical structure through additional attention layers, tree encoders, or full fine-tuning, the proposed framework treats grammar as an explicit inductive bias rather than a learnable module, resulting in a very lightweight model that delivers better performance on edge devices

Figures & tables

Label	Description
GEO	Geographical Entity
ORG	Organization
PER	Person
GPE	Geopolitical Entity
TIM	Time Indicator
ART	Artifact

Table 1: Named Entity Categories Used in the GMB Dataset

Figure 1: Classification dataset class distribution

Figure 2: Classification dataset text data length distribution

Figure 3: NER dataset sentence length distribution

Figure 4: Model diagram of the proposed system

Feature Group	Description	Advantage
POS Ratios	Ratio of different part-of-speech tags (nouns, verbs, adjectives, etc.)	Captures syntactic distribution and word-class patterns
Sentence Length	Number of words or tokens per sentence	Indicates structural complexity
Function Words	Frequency of prepositions, conjunctions, articles, pronouns	Reflects grammatical correctness
NP/VP Counts	Count of noun phrases and verb phrases	Captures phrase structure patterns
Clause Count	Number of clauses (main and subordinate) per sentence	Measures syntactic complexity
Adv/Verb Ratio	Ratio of adverbs to verbs	Stylistic and linguistic emphasis

Table 2: Linguistic Feature Groups with Descriptions and Advantages

Figure 5: Transnformer based model architecture with encoder and decoder [ 48 ]

Model	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
LSTM	93.00	89.90	84.10	86.00
BiLSTM	89.80	84.70	77.11	80.74
DBN	91.15	88.12	78.43	82.90
BERT	94.00	90.52	89.59	90.00
XLNET	94.50	90.60	89.25	89.90

Table 3: Performance Metrics of Different Algorithms without grammar feature

Figure 6: Classification model performance without grammar features

Model	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
BiLSTM	96.19	94.69	91.45	93.04
BERT	96.48	93.46	93.91	93.69
XLNet	96.01	95.11	90.29	92.64
LSTM	95.82	94.76	89.94	92.28
DBN	93.27	89.43	85.97	87.66

Table 4: Performance Metrics of Different Models with grammar-based features

Figure 7: Classification model performance with grammar features

Model	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
DBN	91.88	88.80	90.88	90.01
LSTM	85.45	74.10	86.5	78.93
BiLSTM	85.25	72.67	83.25	78.46
BERT	93.29	92.80	90.32	92.73
XLNet	84.53	71.61	88.26	79.43

Table 5: Performance Metrics of NER models without grammatical feature

Figure 8: NER model performance without grammar features

Model	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
DBN	95	94	95	94
LSTM	96	94.1	96.5	93.93
BiLSTM	97	93	94	93.3
BERT	95	96	95	93
XLNet	88.67	87	89	87.48

Table 6: Performance Metrics of NER models with grammatical feature

Figure 9: NER model performance without grammar features

Text Classification

Text classification assigns text to categories such as topic, sentiment, intent, stance, or risk, supporting search, moderation, and decisions.

See all 226 papers

May 23, 2026

Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers

The quadratic complexity of self-attention in Transformer models remains a significant bottleneck for processing long sequences and deploying large language models efficiently. For this approach, there has been significant research into Sparse Attention, and Deepseek Sparse Attention has combined various methods of creating segments of tokens to reduce the time complexity. This paper introduces a novel approach, Grammatically-Guided Sparse Attention, which constrains attention computations based on the grammatical roles of tokens. By leveraging Parts-of-Speech (POS) tags, attention masks are dynamically generated that enforce linguistically coherent connections between tokens, reducing the computational graph without sacrificing essential linguistic dependencies. Two masking strategies are proposed and evaluated: a hard mask that strictly allows only predefined grammatical interactions, and a soft mask that biases attention towards these interactions. The experiments, conducted on the SST-2 sentiment classification task using a DistilBERT-like architecture, demonstrate that Grammatically-Guided Sparse Attention maintains comparable accuracy to full attention while significantly reducing the theoretical computational overhead. Preliminary results show accuracy values of 0.8200 for hard masking and 0.8165 for soft masking, closely matching the 0.8200 of full attention, providing a path towards more efficient, interpretable, and linguistically-informed Transformer architectures.

Spandan Pratyush

Independent Researcher, New Delhi 110070, India

Transformer Interpretability Sparse Attention Attention Mechanisms

Ask PDF

Sep 12, 2026

Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability

Automatic sentence function identification is important for many downstream natural language processing (NLP) applications such as dialogue systems, text-to-speech synthesis, and machine translation. However, benchmark resources for Bangla sentence function classification remain limited. To mitigate this gap, this paper introduces a corpus of 10,000 Bangla sentences, manually annotated into four functional categories, namely declarative, interrogative, imperative, and exclamatory. The corpus is nearly balanced across the four classes, with high annotation reliability reflected by a Fleiss' Kappa of 0.82. Furthermore, we evaluate multiple feature representations, including Bag-of-Words (BoW), TF-IDF, and Word2Vec, with several classical machine learning classifiers. In addition, two heterogeneous ensemble models, namely Single-Level Ensemble (SLE) and Double-Level Ensemble (DLE), are utilized to improve classification performance. Experimental results show that TF-IDF consistently outperforms Word2Vec, likely due to its ability to emphasize discriminative lexical cues associated with sentence functions, particularly given the relatively small corpus used to train Word2Vec. The DLE model with TF-IDF features achieves the best performance with accuracy and macro-F1 of 0.95, demonstrating the effectiveness of sparse lexical representations and heterogeneous ensemble learning for this task. Further cross-validation confirms the robustness of the approach, while LIME-based interpretability provides insights into model predictions. The developed corpus and model benchmarking establish strong baselines for Bangla sentence function classification.

Swapnil Kundu Argha, Abdullah Al Shafi, Rowzatul Zannat +4

Department of CSE, Khulna University of Engineering & Technology, Bangladesh · Institute of ICT, Khulna University of Engineering & Technology, Bangladesh · Department of CSE, Daffodil International University, Bangladesh +1

Bengali NLP Low-Resource Language Processing

Ask PDF

Aug 25, 2026

The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

Transformer representations describe trajectories through high-dimensional vector spaces, which are shaped dynamically as tokens incorporate relational context across layers. Such data tend to concentrate on lower-dimensional sub-manifolds, a form of compression quantified by the Intrinsic Dimensionality (ID), the minimum number of independent variables needed to represent them without significant information loss. In this work, we ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold. To this end: (1) We investigate the layer-wise evolution of ID, finding that closed-class items expand earlier and collapse sooner than open-class ones; (2) We show its expansion and contraction to be explained by changes in the neighborhood structure, and hence in the relations between words within a sentence; (3) We compare encoders (ModernBERT, bigbird-roberta-large) and decoders (gemma-2-2B, Llama-3.2-3B), finding that the two families evolve differently across layers, consistently with how each integrates context;(4) We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.

Samuele Vallisa, Federico Ravenda, Claudio Palominos +5

Grammar and Cognition Lab, Department of Translation & Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain. · Euler Institute, Università della Svizzera italiana, Lugano, Switzerland. · Department of Adult Psychiatry and Psychotherapy, University Hospital of Psychiatry Zurich, University of Zurich, Zurich, Switzerland. +2

Representation Geometry in Language Models Transformer Intrinsic Dimensionality

Ask PDF

Abstract

Text Classification

Explore similar work

Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers

Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability

The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers