Embedding Consistency Regulation for Persistent Semantic Routing in Compact Language Models
Organizations: YVIC Research Lab
Abstract
Language models are increasingly deployed under tight memory and compute constraints, where semantic routes (e.g., language, tone, or intent) may be assigned before decoding. However, these routes may not remain reflected in generated responses. We study this phenomenon as persistent semantic routing. We investigate this property using Embedding Consistency Regulation (ECR) as a minimal probe. ECR derives a short discrete prefix for each input using a frozen embedding model and applies the same prefix construction during both training and inference. In controlled prefix-assignment experiments, ECR achieves 99.93% 0.12% language routing accuracy, while random, shuffled, and dummy controls remain near chance (34--36%). This gap is not reliably captured by likelihood: some controls achieve competitive NLL despite losing routing consistency. We introduce Response Routing Recoverability (RRR), an encoder-independent diagnostic that tests whether assigned routes remain observable after generation. Routing recoverability persists across multilingual settings, unsupervised partitions, reduced precision, and transfer tasks. These findings establish routing persistence as a distinct and measurable property of language models, complementary to task performance and likelihood.
Figures & tables
| Model | English | Chinese | Hindi |
|---|---|---|---|
| 3B BF16 | 2.94 | 3.04 | 2.30 |
| 3B FP32 | 2.07 | 2.08 | 1.59 |
| 1B BF16 | 3.14 | 3.33 | 2.49 |
| 1B BF16 + ECR | 3.44 | 3.59 | 2.75 |
| 1B FP32 | 3.59 | 3.50 | 2.76 |
| 1B FP32 + ECR | 2.76 | 2.79 | 2.12 |
| Model | E5 | LaBSE | mMPNet | mMiniLM |
|---|---|---|---|---|
| 3B BF16 | 0.8455 | 0.6794 | 0.5871 | 0.5423 |
| 3B FP32 | 0.8647 | 0.7315 | 0.6479 | 0.6122 |
| 1B BF16 | 0.8353 | 0.6569 | 0.5296 | 0.4866 |
| 1B BF16 + ECR | 0.8371 | 0.6637 | 0.5223 | 0.4835 |
| 1B FP32 | 0.8372 | 0.6707 | 0.5259 | 0.5010 |
| 1B FP32 + ECR | 0.8558 | 0.7091 | 0.6078 | 0.5772 |
| Model | Intra | Inter | Ratio |
|---|---|---|---|
| 3B BF16 | 42.51 | 43.54 | 0.976 |
| 3B FP32 | 39.20 | 41.05 | 0.955 |
| 1B BF16 | 48.02 | 48.95 | 0.981 |
| 1B BF16 + ECR | 41.17 | 42.48 | 0.969 |
| 1B FP32 | 46.51 | 47.33 | 0.983 |
| 1B FP32 + ECR | 39.66 | 41.91 | 0.946 |
| Run | Lang. Acc. | NLL | OSim(E5) |
|---|---|---|---|
| ECR | 0.9993 0.0012 | 6.12 | 0.897 0.0003 |
| Random | 0.3430 0.0030 | 6.11 | 0.858 0.0003 |
| Shuffled | 0.3640 0.0080 | 6.11 | 0.858 0.0004 |
| Dummy | 0.3553 0.0059 | 6.14 | 0.856 0.0014 |
| RRR@1 | RRR@2 | |||||||
| Metric | Teacher | ECR | Random | Shuffled | Teacher | ECR | Random | Shuffled |
| L | 86.85 | 92.07 ± 0.13 | 34.22 ± 1.08 | 32.17 ± 0.29 | 97.03 | 98.25 ± 0.11 | 66.40 ± 1.09 | 65.56 ± 0.36 |
| E1 | 14.75 | 13.58 ± 0.45 | 4.80 ± 0.28 | 1.96 ± 0.23 | 28.50 | 25.41 ± 0.62 | 9.88 ± 0.61 | 6.29 ± 0.06 |
| E2 | 12.80 | 11.18 ± 0.18 | 4.93 ± 0.14 | 2.09 ± 0.18 | 23.47 | 20.22 ± 0.18 | 10.07 ± 0.53 | 6.13 ± 0.26 |
| I1 | 19.23 | 19.94 ± 0.64 | 3.44 ± 0.35 | 0.57 ± 0.16 | 32.42 | 32.45 ± 0.45 | 6.77 ± 0.24 | 2.02 ± 0.15 |
| I2 | 10.65 | 7.65 ± 0.30 | 3.49 ± 0.25 | 0.61 ± 0.10 | 19.95 | 14.96 ± 0.39 | 7.15 ± 0.43 | 2.63 ± 0.27 |
| RRR@1 | ||||||
| Prec. | Partition | NLL | OSim | Teacher | ECR | Chance |
| BF16 | (SFT) | 5.623 | 0.9030 | – | – | – |
| 5.642 | 0.9035 | 47.38 | 32.24 | 3.13 | ||
| 5.652 | 0.9038 | 53.48 | 23.51 | 0.39 | ||
| FP32 | (SFT) | 2.738 | 0.8746 | – | – | – |
| 2.747 | 0.8751 | 70.23 | 34.74 | 25.00 | ||
| Backend | Format | |||
| HuggingFace | FP32 | 34.74 | 21.68 | 11.94 |
| FP16 | 34.49 | 21.70 | 11.73 | |
| GGUF | Q6 | 28.48 | 29.54 | 16.23 |
| Q5 | 32.42 | 28.88 | 15.13 | |
| Q4 | 48.51 | 20.42 | 13.29 | |
| Q3 | 38.21 | 11.36 | 6.84 |
| RRR@1 | RRR@2 | |||||||
|---|---|---|---|---|---|---|---|---|
| Partition | NLL | OSim(E5) | Teacher | ECR | Random | Teacher | ECR | Random |
| 3.611 0.128 | 0.806 0.003 | – | – | – | – | – | – | |
| 3.566 0.071 | 0.811 0.007 | 31.65 | 20.22 0.57 | 25.00 | 61.05 | 56.15 0.58 | 50.00 | |
| 3.645 0.051 | 0.809 0.005 | 24.25 | 13.53 0.73 | 12.50 | 41.75 | 29.65 0.61 | 25.00 | |
| 3.521 0.039 | 0.800 0.007 | 23.75 | 10.27 0.93 | 6.25 | 38.15 | 19.40 1.35 | 12.50 | |
| 3.531 0.031 | 0.796 0.005 | 25.65 | 6.30 0.41 | 3.12 | 37.40 | 12.67 1.11 | 6.25 | |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Train | Test | Avg. Length |
|---|---|---|---|
| English | 30k | 3.4k | 124.6 |
| Chinese | 30k | 3.3k | 138.0 |
| Hindi | 30k | 3.3k | 172.2 |
| Total | 90k | 10k | 144.9 |
| Field | Description |
|---|---|
| L | Language of the answer |
| E1 | Primary emotion label ( ) |
| E2 | Secondary emotion label ( ) |
| I1 | Primary intent label ( ) |
| I2 | Secondary intent label ( ) |
| Q | Query text (single-turn instruction) |
| Field | Description |
|---|---|
| dialog_id | Dialogue identifier |
| e1 | Primary emotion label |
| e2 | Secondary emotion label |
| i1 | Primary intent label |
| i2 | Secondary intent label |
| EN_Q | English query |
| Item | Specification |
|---|---|
| Model name | Luna-0.6B |
| Package size | 693 MB |
| Training data | 875,596 dialogue fragments |
| Throughput | 15 tokens/s (iPhone) |
| Core abilities | summarization; translation; offline RAG; sentiment analysis |
| Item | Specification |
|---|---|
| Device | iPhone 14 (iOS 26) |
| Backend | llama.cpp with Metal acceleration |
| Model format | GGUF (Q4_KM) |
| Runtime footprint | 693 MB (model + RAG index + runtime) |
| Peak RAM usage | 1.27 GB |
| Throughput | 14–15 tokens/s (offline) |
| Index Type | HNSW, 64-d |
| Index Size | 48.9 MB |
| RAG Records | 381,943 text entries |
| Query Latency | ms/query |
| Top- | 5 |
| Similarity Metric | Cosine distance |
LLM Routing
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
See all 380 papers