cs.CLNov 14, 2026

A Comparative Evaluation of BERT-Based and LLM-Based Models for Indic Sentence Similarity

Authors: Aishwarya Gunjal, Kanchan Khemnar, Pawan Poojary, Pranav Kulkarni, Raviraj Joshi

Organizations: Pune Institute of Computer Technology, Pune, India · L3Cube Labs, Pune, India

Abstract

The relative effectiveness of BERT-based encoders and large language model (LLM)-based embeddings for Indic semantic textual similarity remains insufficiently characterized. We compare eight BERT-based and five LLM-based models on the 1,500-pair STS Benchmark development set translated into ten Indic languages and on the native Marathi MahaSTS test set. IndicSBERT achieves the highest Pearson correlation in all ten translated language sets, with a macro-average of 0.843, followed by BGE-M3 at 0.792. Harrier-270M is the strongest LLM-based model on translated STS-B, with a macro-average of 0.751. On MahaSTS, IndicSBERT and Bhasha-Embed-v0 achieve Pearson correlations of 0.901 and 0.900, respectively, while EmbeddingGemma leads the LLM-based models at 0.871. The highest-scoring BERT-based model thus exceeds the highest-scoring LLM-based model in both settings, although the magnitude of the difference varies across benchmarks. These findings indicate that LLM-derived embeddings do not uniformly improve Indic sentence similarity over established encoders and motivate evaluation that accounts for language coverage, task supervision, and benchmark construction.

Figures & tables

Explore similar work

CardsList
  1. Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

    Jul 5, 2026Lucas Hideki Takeuchi Okamura, Alexandre Alcoforado, Anna Helena Reali CostaSentence EmbeddingsText Embedding Evaluation

  2. Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs

    Oct 19, 2026Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker +24Language ModelingLanguage Model Pretraining

  3. NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

    Oct 13, 2026Gaurab Baral, Mimansak Nepal, Grishma BasnetSentiment AnalysisLLM Evaluation