StaRSE: Compact Russian Sentence Embeddings with a Sign-Coded Static Encoder
Organizations: ITMO University, Saint Petersburg, Russia
Abstract
Transformer sentence encoders provide strong retrieval quality, but their inference cost becomes a bottleneck when embeddings must be produced on CPUs or refreshed frequently. We introduce StaRSE, a compact Russian static sentence encoder that removes Transformer inference at deployment time: token directions are stored as sign bits, token norms are stored separately, and sentences are encoded by mean pooling over norm-scaled sign directions. StaRSE is initialized from ruBERT-base, fine-tuned on Russian adjacent-sentence and task-shaped pairs, and post-trained with a contrastive stage that directly optimizes the sign-projected token table. On RuMTEB, StaRSE reaches a task mean of , points above the strongest included static baseline. The packaged model occupies MiB and has a mean CPU latency of ms per sentence, compared with an estimated MiB 16-bit parameter footprint and ms mean CPU latency for the closest-quality contextual encoder. Ablations examine adjacent-sentence supervision, token norms, and direct sign-projected post-training.
Figures & tables
| Model (dims) | Mean | Class. | Cluster. | Multilabel | PairClass. | Rerank | Retrieval | STS |
| Sentence Transformers (STs) | ||||||||
| ruBERT-tiny-turbo (312) | 53.76 | 55.08 | 49.62 | 32.05 | 56.27 | 54.94 | 48.82 | 71.73 |
| ruBERT-mini-FRIDA (312) | 63.22 | 64.28 | 59.71 | 47.19 | 60.36 | 65.62 | 64.70 | 72.14 |
| BERTA (768) | 69.39 | 71.22 | 64.78 | 56.46 | 65.74 | 69.72 | 73.79 | 73.72 |
| FRIDA (1536) | 70.95 | 73.71 | 67.49 | 52.23 | 66.45 | 71.50 | 76.38 | 74.35 |
| GigaEmbeddings (2048) | 74.16 | 76.66 | 67.48 | 57.07 | 79.57 | 74.02 | 81.41 | 75.75 |
| Model (dims) | nDCG@ |
| Sentence Transformers (STs) | |
| USER2-small (384) | 51.69 |
| USER2-base (768) | 54.17 |
| USER-bge-m3 (1024) | 58.53 |
| KaLM-v1.5 (896) | 53.75 |
| jina-embeddings-v3 (1024) | 49.67 |
| Max doc tokens | BM25 | StaRSE | StaRSE+BM25 | USER2-small |
| 128 | 28.34 | 32.11 | 30.84 | 42.31 |
| 512 | 30.09 | 33.63 | 34.17 | 46.97 |
| 1024 | 32.47 | 36.88 | 36.05 | 48.22 |
| 2048 | 34.07 | 40.90 | 38.38 | 48.67 |
| 4096 | 68.12 | 52.01 | 69.93 | 53.70 |
| 8192 | 69.97 | 51.66 | 73.07 | 51.69 |
| Model (dims) | Size (MiB) | RuMTEB | CPU (ms) | GPU (ms) |
| Sentence Transformers (STs) | ||||
| ruBERT-tiny-turbo (312) | 55.7 | 53.76 | 1.43 | 1.68 |
| ruBERT-mini-FRIDA (312) | 61.5 | 63.22 | 2.97 | 3.01 |
| BERTA (768) | 245 | 69.39 | 19.36 | 3.80 |
| FRIDA (1536) | 1571 | 70.95 | 110.87 | 9.61 |
| GigaEmbeddings (2048) | 6580 | 74.16 | 773.61 | 43.26 |
| Tokenizer | Vocab | Valid tok/word | BM25 nDCG@ |
| ai-forever/ruBert-base | 120,138 | 1.342 | 52.46 |
| deepvk/USER-bge-m3 | 46,166 | 1.696 | 51.02 |
| cointegrated/rubert-tiny2 | 83,828 | 1.303 | 50.81 |
| DeepPavlov/rubert-base-cased-conversational | 100,792 | 1.280 | 50.57 |
| cointegrated/rubert-tiny | 29,564 | 2.171 | 50.27 |
| deepvk/USER2-small | 50,368 | 2.153 | 49.88 |
| Initialization source | PCA | STS dev |
| Input embeddings | token-level | 0.76810 |
| Input embeddings | sentence-level | 0.73663 |
| Averaged contextual embeddings | sentence-level | 0.73349 |
| Per-token last-hidden state (Model2Vec) | token-level | 0.64753 |
| Variant | RuSTS dev | RuMTEB |
| Distillation baselines | ||
| BERTA-distilled static student | 74.7 | 42.0 |
| USER2-small-distilled static student | 69.2 | 44.2 |
| Positive-pair construction | ||
| Self-pairs | 71.1 | 37.2 |
| Raw adjacent Cultura | 76.1 | 48.2 |
| Setting | RuMTEB | |
| Training route to a sign-coded token table | ||
| Post-hoc signing after contrastive fine-tuning; no post-training | 49.33 | – |
| Contrastive continuation, then post-hoc signing | 50.09 | +0.76 |
| Optimized sign-projected post-training, final StaRSE recipe | 51.16 | +1.83 |
| Inference-time negative controls on the final checkpoint | ||
| StaRSE inference: token norms kept; real-valued pooled sentence vector | 51.16 | – |
Sentence Embeddings
Sentence embeddings map sentences to vectors for semantic similarity, retrieval, clustering, and downstream language tasks.
See all 41 papers