Explainable Semantic Textual Similarity via Dissimilar Span Detection

Diego Miguel Lozano; Daryna Dementieva; Alexander Fraser

cs.CLMar 22, 2026

Explainable Semantic Textual Similarity via Dissimilar Span Detection

Authors: Diego Miguel Lozano, Daryna Dementieva, Alexander Fraser

Organizations: School of Computation, Information and Technology Technical University of Munich · Munich Center for Machine Learning (MCML)

Abstract

Semantic Textual Similarity (STS) is a crucial component of many Natural Language Processing (NLP) applications. However, existing approaches typically reduce semantic nuances to a single score, limiting interpretability. To address this, we introduce the task of Dissimilar Span Detection (DSD), which aims to identify semantically differing spans between pairs of texts. This can help users understand which particular words or tokens negatively affect the similarity score, or be used to improve performance in STS-dependent downstream tasks. Furthermore, we release a new dataset suitable for the task, the Span Similarity Dataset (SSD), developed through a semi-automated pipeline combining large language models (LLMs) with human verification. We propose and evaluate different baseline methods for DSD, both unsupervised, based on LIME, SHAP, LLMs, and our own method, as well as an additional supervised approach. While LLMs and supervised models achieve the highest performance, overall results remain low, highlighting the complexity of the task. Finally, we set up an additional experiment that shows how DSD can lead to increased performance in the specific task of paraphrase detection.

Figures & tables

Figure 1: Examples of the Dissimilar Span Detection task, i.e., given a pair of texts, identify which spans differ semantically. Here, we show two pairs that contain dissimilar spans and one pair that does not. Cosine similarities are calculated with the Sentence Transformer model all-MiniLM-L6-v2 . Note that pairs containing dissimilar spans might still yield a high semantic textual similarity, sometimes even higher than sentences that are equivalent in meaning. Therefore, relying on STS alone could potentially lead to erroneous decisions in downstream tasks.

Sentence 1	Sentence 2	Span Similarity	Sentence Similarity
It only utilizes {{a few refineries}} in the Northeast.	It only utilizes {{numerous facilities}} in the Northeast.	0	0
It was {{restored}} in the {{1980s}} .	It was {{destroyed}} in the {{eighties}} .	0,1	0
Thank you for {{wasting my money}} .	Thank you for {{misusing my funds}} .	1	1
There are depots at {{Quilpie}} and {{Roma}} .	There are depots at {{Brisbane}} and {{Sydney}} .	0,0	0

Table 1: Examples of shorter sentences from the SSD. Spans are denoted using double curly braces, and then annotated respectively with a 0, in case the span pair differs in meaning, or 1, if they are equivalent. If there exist several annotated spans, these values are separated by a comma. Finally, the sentence similarity column denotes whether the meaning of the sentence pair as a whole is equivalent in meaning (1) or not (0). We have highlighted similar (green) and dissimilar spans (red) for better readability.

Attribute	Value
Sentence pairs	1,000
Span pairs	1,296
Dissimilar span pairs	648 (50 %)
Equivalent span pairs	648 (50 %)
Dissimilar sentence pairs	571 (57 %)
Equivalent sentence pairs	429 (43 %)

Table 2: Global statistics of the SSD. For the word counts, simple whitespace tokenization was used, not considering punctuation.

Agreement	Inter-annotator	Annotator-dataset
Span boundaries	0.74 (0.05)	0.67 (0.03)
Span labels	0.87 (0.05)	0.91 (0.04)
Span count	0.66 (0.06)	0.61 (0.07)

Table 3: Mean Cohen’s

\kappa

scores on different annotation aspects (standard deviations in parenthesis).

Figure 2: Example of trigram replacements. The considered trigrams proceeding from the first sentence (highlighted) are inserted into the second sentence, replacing the original trigrams.

		SSD				SemEval-2016
Method	Model Size	F1-Global	F1-NoDiff	F1-Diff	Eval. Time	F1-Diff	Eval. Time
LIME all-mpnet-base-v2 (0.001)	109M	0.463	0.782	0.223	1981.81	0.109	199.89
SHAP all-MiniLM-L6-v2 (0.010)	22.7M	0.366	0.434	0.131	44.47	0.306	4.52
Embedding all-mpnet-base-v2 (0.006)	109M	0.469	0.529	0.423	11.28	0.352	1.13
Embedding text-embedding-004 (0.005)	Billions	0.547	0.666	0.459	611.42	0.338	34.57
LLM Claude 3.5 Sonnet	Billions	0.750	0.895	0.640	337.80	0.389	39.22

Table 4: Summary of the results. LLMs and supervised methods (Token-Classification-DSD) perform best, followed by our proposed method (Embedding-DSD). For methods that require a threshold, we include it next to the model’s name. Evaluation times are reported in minutes. For experiments relying on external APIs ( text-embedding-004 and Claude 3.5 Sonnet), times might vary depending on rate limits, connection speeds, etc. For these models, the exact number of parameters remains undisclosed, so we are unable to report it. The rest of experiments were run on a NVIDIA RTX 3080Ti graphics card.

Model	STS	STS+DSD
all-MiniLM-L6-v2 (0.65, 8 $\cdot$ 10 -3 )	0.720	0.808
all-mpnet-base-v2 (0.63, 8 $\cdot$ 10 -3 )	0.795	0.868

Table 5: Comparison of accuracies on the PAWS-Wiki Labeled using uniquely STS or combining STS with DSD. We include the STS and DSD thresholds in parenthesis next to the models.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

		NoDiff			Time
Method	Model	Precision	Recall	F1	(minutes)
LIME	all-MiniLM-L6-v2 (0.010)	0.253 (0.010)	0.395 (0.029)	0.269 (0.013)	120.88
LIME	all-mpnet-base-v2 (0.001)	0.154 (0.017)	0.093 (0.011)	0.109 (0.011)	199.89
SHAP	all-MiniLM-L6-v2 (0.005)	0.247 (0.012)	0.529 (0.025)	0.306 (0.016)	4.52
SHAP	all-mpnet-base-v2 (0.001)	0.243 (0.016)	0.480 (0.022)	0.293 (0.019)	21.58
Embedding	all-MiniLM-L6-v2 (0.005)	0.249 (0.015)	0.842 (0.051)	0.362 (0.019)	0.38

Appendix

Table 6: Results of the different methods on the SemEval-2016 Task 2 data. As previously mentioned, we only consider the sentences that contain spans annotated as opposite . That is why we only report the NoDiff scores. We include standard errors within parenthesis. For methods that require a threshold, its value is included in parenthesis next to the model’s name. Best results from each method are shown in bold, while best results overall in a column are also underlined. The time column reflects the total time to run the entire evaluation (i.e., 5 folds). Note that for methods that rely on external APIs (e.g., LLM-DSD), times might fluctuate depending on rate limits, connection speeds, etc.

Sentence 1	Sentence 2	Method	Annotation
Following the Civil War, the plantation house was destroyed by hurricanes.	Before the Civil War, hurricanes destroyed the plantation house.	Reference Annotation	{{Before the Civil War}} , hurricanes destroyed the plantation house.
		LIME all-MiniLM-L6-v2 $\alpha=\text{0.030}$	{{Before the}} Civil {{War}} , hurricanes destroyed the {{plantation house.}}
		SHAP all-MiniLM-L6-v2 $\alpha=\text{0.010}$	Before the Civil War {{,}} hurricanes destroyed the plantation house {{.}}
		Embedding all-MiniLM-L6-v2 $\alpha=\text{0.010}$	Before the Civil War, hurricanes destroyed the plantation house.
		Embedding text-embedding-3-large $\alpha=\text{0.005}$	{{Before}} the Civil War, hurricanes destroyed the plantation house.
		LLM GPT-4o	Before {{the Civil War}} , hurricanes destroyed the plantation house.

Appendix

Table 8: Examples of annotated outputs. The first two sentence pairs are coming from the SSD, while the third proceeds from the SemEval-2016 Task 2 data. In the latter, it can be seen that the differing dates “Dec. 4” and “Jan. 24” have not been included in the reference annotations (they were annotated as similar ). This is one of the caveats with the SemEval-2016 Task dataset: there are no unambiguous annotations of dissimilarity beyond the opposite spans.

indices,\,repl\in replacements

Appendix

Algorithm 1 Embedding-DSD.

nGramSize\leftarrow 1,\,\text{len}(sent2Unigrams)

Appendix

Algorithm 2 Calculate all possible n -gram replacements.

Dataset	License	Homepage
Span Similarity Dataset (SSD)	CC BY-SA 4.0	https://dmlls.github.io/dissimilar-span-detection
SemEval-2016 Task 2 Dataset	CC BY-SA 4.0	https://alt.qcri.org/semeval2016/task2

Appendix

Table 9: Licenses of the datasets we used in our work.

Model	License	Homepage
all-MiniLM-L6-v2	Apache License 2.0	https://huggingface.co/sentence- transformers/all-MiniLM-L6-v2
all-mpnet-base-v2	Apache License 2.0	https://huggingface.co/sentence- transformers/all-mpnet-base-v2
text-embedding-3-large	Proprietary	https://platform.openai.com/docs/guides/embeddings/#embedding-models
text-embedding-004	Proprietary	https://ai.google.dev/gemini-api/docs/embeddings
GPT-4o	Proprietary	https://openai.com/index/hello-gpt-4o
Llama 3.3 70B Instruct	Llama 3.3 Community License Agreement	https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_3/

Appendix

Table 10: Licenses of the models we used in our work.

Semantic Textual Similarity

Semantic Textual Similarity measures how closely two texts match in meaning, supporting comparison, retrieval, clustering, and evaluation.

See all 44 papers

DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity
May 28, 2026Kaijie Zheng, Weiqin Wang, Yile Wang +1Semantic Textual Similarity
College of Computer Science and Software Engineering, Shenzhen University
High-Level Text Preprocessing for Semantic Similarity Analysis of Discursive Texts: A Framework and Empirical Demonstration
Sep 27, 2026Mehmet Murat Albayrakoglu, Mehmet Nafiz AydinSemantic Textual Similarity
Işık University Faculty of Economics, Administrative and Social Sciences Department of Management Information Systems Istanbul, Türkiye · Boğaziçi University Faculty of Managerial Sciences, Department of Management Information Systems Istanbul, Türkiye
MATCHA: Matching Text via Contrastive Semantic Alignment
May 26, 2026Siran Li, Ece Sena Etoglu, Carsten Eickhoff +1LLM Evaluation Semantic Textual Similarity
University of Tübingen

May 28, 2026J/K move · Enter open · S save

DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity

Kaijie Zheng, Weiqin Wang, Yile Wang, Hui Huang

College of Computer Science and Software Engineering, Shenzhen University

Semantic Textual Similarity

Open paper AskPDF arXiv

Calculating semantic textual similarity is a foundational task in natural language processing. Current large language models (LLMs) based methods typically rely on extracting last-layer hidden states with fixed dimensions to compute similarity for every text pairs. We argue that this paradigm is suffer from two limitations: (i) The last hidden layer encodes more general knowledge rather than just semantic knowledge, making it suboptimal for semantic similarity computation; (ii) The hidden layer dimensions of LLMs are generally very large, which introduces some redundancy and noise for representing semantics. In this work, we propose DySem, a novel training-free framework that investigates more semantic-related internal components of LLMs via multilingual consensus, and shifts away from static representation spaces in favor of dynamic, sample-specific semantic dimensions by constructing text-dependent joint semantic set and computes similarity over this shared dimensional subset. Extensive experiments across various LLMs show that our method consistently outperforms recent baselines while maintaining lower dimensions for similarity calculation. The code is released at https://github.com/szu-tera/DySem.

Abstract

Semantic Textual Similarity

Explore similar work

DySem: Uncovering Dynamic Semantic Components of Large Language Models for Calculating Semantic Textual Similarity

High-Level Text Preprocessing for Semantic Similarity Analysis of Discursive Texts: A Framework and Empirical Demonstration

MATCHA: Matching Text via Contrastive Semantic Alignment