cs.CLNov 10, 2026

StaRSE: Compact Russian Sentence Embeddings with a Sign-Coded Static Encoder

Authors: Boris Malashenko, Ivan Jarsky, Egor Soldatov, Vladislav Luzhnov, Sviatoslav Stumpf, Vladislav Shmatkov, Viacheslav Shalamov, Andrey Stankevich, 

Organizations: ITMO University, Saint Petersburg, Russia

Abstract

Transformer sentence encoders provide strong retrieval quality, but their inference cost becomes a bottleneck when embeddings must be produced on CPUs or refreshed frequently. We introduce StaRSE, a compact Russian static sentence encoder that removes Transformer inference at deployment time: token directions are stored as sign bits, token norms are stored separately, and sentences are encoded by mean pooling over norm-scaled sign directions. StaRSE is initialized from ruBERT-base, fine-tuned on Russian adjacent-sentence and task-shaped pairs, and post-trained with a contrastive stage that directly optimizes the sign-projected token table. On RuMTEB, StaRSE reaches a task mean of 51.1651.16, 7.987.98 points above the strongest included static baseline. The packaged model occupies 11.311.3 MiB and has a mean CPU latency of 0.080.08 ms per sentence, compared with an estimated 55.755.7 MiB 16-bit parameter footprint and 1.431.43 ms mean CPU latency for the closest-quality contextual encoder. Ablations examine adjacent-sentence supervision, token norms, and direct sign-projected post-training.

Figures & tables

Explore similar work

CardsList
  1. Polish ModernBERT: The Long and Short of Polish Language Understanding

    Sep 1, 2026Michał Perełkiewicz, Sławomir Dadas, Rafał Poświata +1Language Model PretrainingLong-Context Language Model Evaluation

  2. Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

    Aug 24, 2026Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev +1Text Embedding ModelsMixture-of-Experts Models

  3. AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

    Aug 8, 2026Ahmed Amine Aliane, Hassina Aliane, Nasredine SemmarLanguage Model PretrainingArabic NLP