Language models are increasingly deployed under tight memory and compute constraints, where semantic routes (e.g., language, tone, or intent) may be assigned before decoding. However, these routes may not remain reflected in generated responses. We study this phenomenon as persistent semantic routing. We investigate this property using Embedding Consistency Regulation (ECR) as a minimal probe. ECR derives a short discrete prefix for each input using a frozen embedding model and applies the same prefix construction during both training and inference. In controlled prefix-assignment experiments, ECR achieves 99.93% ± 0.12% language routing accuracy, while random, shuffled, and dummy controls remain near chance (34--36%). This gap is not reliably captured by likelihood: some controls achieve competitive NLL despite losing routing consistency. We introduce Response Routing Recoverability (RRR), an encoder-independent diagnostic that tests whether assigned routes remain observable after generation. Routing recoverability persists across multilingual settings, unsupervised partitions, reduced precision, and transfer tasks. These findings establish routing persistence as a distinct and measurable property of language models, complementary to task performance and likelihood.
Figures & tables
Figure 1: Embedding Consistency Regulation (ECR). Training and inference inputs are aligned through a shared semantic projection and control-prefix construction pipeline. The resulting prefix acts as a reusable semantic anchor for compact models under fixed-capacity deployment.
Figure 2: Overview of ECR. Queries are projected into a fixed semantic space and serialized into discrete control prefixes shared across training and inference.
Model
English
Chinese
Hindi
3B BF16
2.94
3.04
2.30
3B FP32
2.07
2.08
1.59
1B BF16
3.14
3.33
2.49
1B BF16 + ECR
3.44
3.59
2.75
1B FP32
3.59
3.50
2.76
1B FP32 + ECR
2.76
2.79
2.12
Table 1: Negative log-likelihood (NLL).
Model
E5
LaBSE
mMPNet
mMiniLM
3B BF16
0.8455
0.6794
0.5871
0.5423
3B FP32
0.8647
0.7315
0.6479
0.6122
1B BF16
0.8353
0.6569
0.5296
0.4866
1B BF16 + ECR
0.8371
0.6637
0.5223
0.4835
1B FP32
0.8372
0.6707
0.5259
0.5010
1B FP32 + ECR
0.8558
0.7091
0.6078
0.5772
Table 2: Output embedding similarity (OSim) between reference answers A and generated outputs A^ .
Model
Intra ↓
Inter
Ratio ↓
3B BF16
42.51
43.54
0.976
3B FP32
39.20
41.05
0.955
1B BF16
48.02
48.95
0.981
1B BF16 + ECR
41.17
42.48
0.969
1B FP32
46.51
47.33
0.983
1B FP32 + ECR
39.66
41.91
0.946
Table 3: Representation geometry metrics.
Run
Lang. Acc.
NLL
OSim(E5)
ECR
0.9993 ± 0.0012
6.12
0.897 ± 0.0003
Random
0.3430 ± 0.0030
6.11
0.858 ± 0.0003
Shuffled
0.3640 ± 0.0080
6.11
0.858 ± 0.0004
Dummy
0.3553 ± 0.0059
6.14
0.856 ± 0.0014
Table 4: Prefix assignment controls averaged over three seeds under 1B FP32.
Figure 3: Response Routing Recoverability (RRR). Generated responses are embedded with an external encoder and compared to response centroids for each routing slot. RRR measures whether the originally assigned route ranks among the nearest centroids after generation.
RRR@1
RRR@2
Metric
Teacher
ECR
Random
Shuffled
Teacher
ECR
Random
Shuffled
L
86.85
92.07 ± 0.13
34.22 ± 1.08
32.17 ± 0.29
97.03
98.25 ± 0.11
66.40 ± 1.09
65.56 ± 0.36
E1
14.75
13.58 ± 0.45
4.80 ± 0.28
1.96 ± 0.23
28.50
25.41 ± 0.62
9.88 ± 0.61
6.29 ± 0.06
E2
12.80
11.18 ± 0.18
4.93 ± 0.14
2.09 ± 0.18
23.47
20.22 ± 0.18
10.07 ± 0.53
6.13 ± 0.26
I1
19.23
19.94 ± 0.64
3.44 ± 0.35
0.57 ± 0.16
32.42
32.45 ± 0.45
6.77 ± 0.24
2.02 ± 0.15
I2
10.65
7.65 ± 0.30
3.49 ± 0.25
0.61 ± 0.10
19.95
14.96 ± 0.39
7.15 ± 0.43
2.63 ± 0.27
Table 5: Response Routing Recoverability (RRR@K) averaged over three random seeds and four external embedding models (E5, LaBSE, mMPNet, and mMiniLM).
RRR@1
Prec.
Partition
NLL
OSim
Teacher
ECR
Chance
BF16
C=1 (SFT)
5.623
0.9030
–
–
–
C=32
5.642
0.9035
47.38
32.24
3.13
C=256
5.652
0.9038
53.48
23.51
0.39
FP32
C=1 (SFT)
2.738
0.8746
–
–
–
C=4
2.747
0.8751
70.23
34.74
25.00
Table 6: Unsupervised partitioning results. RRR@1 is averaged across four external encoders. Chance denotes random recovery ( 1/C ).
Backend
Format
C=4
C=32
C=256
HuggingFace
FP32
34.74
21.68
11.94
FP16
34.49
21.70
11.73
GGUF
Q6
28.48
29.54
16.23
Q5
32.42
28.88
15.13
Q4
48.51
20.42
13.29
Q3
38.21
11.36
6.84
Table 7: RRR@1 across deployment formats derived from the same FP32-trained ECR checkpoint.
RRR@1
RRR@2
Partition
NLL
OSim(E5)
Teacher
ECR
Random
Teacher
ECR
Random
C=1
3.611 ± 0.128
0.806 ± 0.003
–
–
–
–
–
–
C=4
3.566 ± 0.071
0.811 ± 0.007
31.65
20.22 ± 0.57
25.00
61.05
56.15 ± 0.58
50.00
C=8
3.645 ± 0.051
0.809 ± 0.005
24.25
13.53 ± 0.73
12.50
41.75
29.65 ± 0.61
25.00
C=16
3.521 ± 0.039
0.800 ± 0.007
23.75
10.27 ± 0.93
6.25
38.15
19.40 ± 1.35
12.50
C=32
3.531 ± 0.031
0.796 ± 0.005
25.65
6.30 ± 0.41
3.12
37.40
12.67 ± 1.11
6.25
Table 8: FinQA transfer under unsupervised semantic partitioning. Results are averaged over three random seeds and four external embedding models: E5, LaBSE, mMPNet, and mMiniLM.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Language
Train
Test
Avg. Length
English
30k
3.4k
124.6
Chinese
30k
3.3k
138.0
Hindi
30k
3.3k
172.2
Total
90k
10k
144.9
Appendix
Table 9: Dataset statistics.
Field
Description
L
Language of the answer A
E1
Primary emotion label ( e1 )
E2
Secondary emotion label ( e2 )
I1
Primary intent label ( i1 )
I2
Secondary intent label ( i2 )
Q
Query text (single-turn instruction)
Appendix
Table 10: Schema of the processed QA training set.
Field
Description
dialog_id
Dialogue identifier
e1
Primary emotion label
e2
Secondary emotion label
i1
Primary intent label
i2
Secondary intent label
EN_Q
English query
Appendix
Table 11: Field definitions for records in the raw aligned data.
Figure 4: Example records from the aligned multilingual corpus ( raw_data ). Each row contains English, Chinese, and Hindi query–answer pairs with shared emotion and intent labels, illustrating the cross-lingual alignment used to construct QA training pairs.
Figure 5: Representative training trajectories under the no–gradient-clipping setting. The curves illustrate optimization behavior within the evaluated runs rather than strict cross-configuration loss comparability.
Table 13: On-device runtime configuration and resource usage.
Index Type
HNSW, 64-d
Index Size
48.9 MB
RAG Records
381,943 text entries
Query Latency
<2 ms/query
Top- k
5
Similarity Metric
Cosine distance
Appendix
Table 14: Local retrieval subsystem specifications.
Figure 6: Runtime trace of the Luna on-device inference pipeline on iPhone 14. The trace illustrates ECR prefix construction, optional local retrieval, and quantized decoding through the llama.cpp runtime.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.