Text Embedding Evaluation

Ask about this topic

Overview

Text embedding evaluation studies whether vectors produced by language models preserve the distinctions needed for tasks such as semantic similarity, search, clustering, and classification. Researchers use broad suites such as MTEB and BEIR alongside domain- and language-specific benchmarks, controlled contrastive tests, and analyses of embedding geometry and robustness. Results show that model rankings often vary with task, language, prompts, data, and similarity metric, so performance on one benchmark may not predict performance elsewhere. A central challenge is ensuring that benchmark scores reflect meaningful distinctions and real deployment needs, including latency, cost, and the value of combining dense embeddings with lexical retrieval.

Latest papers

CardsList

78 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

  1. Topic Is Not Agenda: A Citation-Community Audit of Text Embeddings

    May 8, 2026Junseon YooText Embedding EvaluationGraph-Based Retrieval

    Pluto Labs

  2. Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus

    May 1, 2026Daria Boratyn, Damian Brzyski, Albert Leśniak +5Semantic Textual SimilarityText Embedding Evaluation

    aJagiellonian Center for Quantitative Political Science, Jagiellonian University, Kraków, Poland · bAGH University, Kraków, Poland · cJagiellonian Center for Digital Humanities, Jagiellonian University, Kraków, Poland

  3. Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings

    Apr 30, 2026Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi +2Text EmbeddingsText Embedding Evaluation

    Tohoku University · The University of Tokyo · Kyoto University +3

  4. SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning

    Apr 23, 2026Hans Ole Hatzel, Ekaterina Artemova, Haimo Paul Stiemer +2Semantic Textual SimilarityRepresentation Learning

    University of Hamburg1 · Toloka AI2 · German UDS3 +1

  5. Finding Meaning in Embeddings: Concept Separation Curves

    Apr 23, 2026Paul Keuren, Marc Ponsen, Robert Ayoub BagheriSentence EmbeddingsText Embedding Evaluation

    Utrecht University · Statistics Netherlands

  6. TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications

    Apr 20, 2026Pranshav Gajjar, Vijay K ShahText Embedding EvaluationText Embedding Models

    NextG Wireless Lab, North Carolina State University, Raleigh, USA

  7. FLARE: Task-agnostic embedding model evaluation through a normalization process

    Apr 19, 2026Jingzhou Jiang, Yixuan Tang, Yi Yang +1Text Embedding EvaluationText Embedding Models

    The Hong Kong University of Science and Technology

  8. JFinTEB: Japanese Financial Text Embedding Benchmark

    Apr 17, 2026Masahiro Suzuki, Hiroki SakajiText Embedding EvaluationText Embedding Models

    Amova Asset Management Co., Ltd. · Tokyo, Japan · Hokkaido University +1

  9. Comparison of Modern Multilingual Text Embedding Techniques for Hate Speech Detection Task

    Apr 16, 2026Evaldas Vaiciukynas, Paulius Danenas, Linas Ablonskis +5Sentence EmbeddingsText Embedding Evaluation

    Department of Information Systems, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania · Centre of Information Systems Design Technologies, Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania

  10. On the Representational Limits of Quantum-Inspired 1024-D Document Embeddings: An Experimental Evaluation Framework

    Apr 10, 2026Dario MaioQuantum-Inspired MLText Embedding Evaluation

    Department of Computer Science and Engineering, University of Bologna, Italy

  11. LMEB: Long-horizon Memory Embedding Benchmark

    Mar 13, 2026Xinping Zhao, Xinshuo Hu, Jiaxin Xu +9Text Embedding EvaluationIR Evaluation

    Harbin Institute of Technology (Shenzhen) · Shenzhen Loop Area Institute (SLAI) · Peking University

  12. The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instruments

    Mar 10, 2026Hubert PlisieckiText EmbeddingsText Embedding Evaluation

    IDEAS Research Institute, Warsaw, Poland

  13. From Word2Vec to Transformers: Text-Derived Composition Embeddings for Filtering Combinatorial Electrocatalysts

    Mar 9, 2026Lei Zhang, Markus StrickerVirtual ScreeningCompositional Representation Learning

    Interdisciplinary Centre for Advanced Materials Simulation, Ruhr University Bochum, Universitätsstraße 150, 44801 Bochum, Germany

  14. GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models

    Dec 17, 2025Bozhou Li, Sihan Yang, Yushuo Guan +6Text Embedding EvaluationT2I Generation

    Peking University · Kling Team, Kuaishou Technology · Xi’an Jiaotong University +1

  15. From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures

    Nov 27, 2025Florian Rottach, William Rudman, Bastian Rieck +2Representation Geometry in Language ModelsText Embedding Evaluation

    University of Tübingen · Boehringer Ingelheim GmbH, Biberach (Riss), Germany · The University of Texas at Austin +1

  16. A Framework for Deductive Semantic Content Analysis at Scale in Science Education Using Text Embeddings

    Aug 27, 2025Jonas Timmann Mjaaland, Markus Fleten Kreutzer, Halvor Tyseng +5Educational TechnologyText Embedding Evaluation

    Center for Interdisciplinary Education, University of Oslo, NO-0316 Oslo, Norway · Laboratory of Atomic and Solid State Physics, Cornell University, Ithaca, NY 14853, USA · Department of Physics, California State University Fullerton, Fullerton, CA 92831, USA +1

  17. ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings

    Aug 3, 2025Ali Shiraee Kasmaee, Mohammad Khodadad, Mahdi Astaraki +4Text Embedding EvaluationInformation Retrieval

    Department of Computational Science and Engineering, McMaster University, Canada · BASF Canada Inc., Canada · BASF Corporation, USA

  18. Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

    Jun 13, 2025Mahmoud Amiri, Thomas BocklitzDocument ChunkingText Embedding Evaluation

    Leibniz Institute of Photonic Technology, Albert-Einstein-Strasse 9, 07745 Jena, Germany · Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich Schiller University Jena, Helmholtzweg 4, 07743 Jena, Germany