Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States

Dianxing Zhang; Gang Li; Sheng Li

cs.AIMar 31, 2026

Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States

Authors: Dianxing Zhang, Gang Li, Sheng Li

Organizations: Digital China AI Research Institute

Abstract

Routing is widely used to scale large language models, from Mixture-of-Experts gating to multi-model/tool selection. A common belief is that routing to a task ``expert'' activates sparser internal computation and thus yields more certain and stable outputs (the Sparsity--Certainty Hypothesis). We test this belief by injecting routing-style meta prompts as a textual proxy for routing signals in front of frozen instruction-tuned LLMs. We quantify (C1) internal density via activation sparsity, (C2) domain-keyword attention, and (C3) output stability via predictive entropy and semantic variation. On a RouterEval subset with three instruction-tuned models (Qwen3-8B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.2), meta prompts consistently densify early/middle-layer representations rather than increasing sparsity; natural-language expert instructions are often stronger than structured tags. Attention responses are heterogeneous: Qwen/Llama reduce keyword attention, while Mistral reinforces it. Finally, the densification--stability link is weak and appears only in Qwen, with near-zero correlations in Llama and Mistral. We present RIDE as a diagnostic probe for calibrating routing design and uncertainty estimation.

Figures & tables

Figure 1: Multi-model overview of $\Delta\mathrm{Hoyer}$ across Early/Middle/Late segments.

Panel of Figure 2 — Figure 2: Multi-model overview of $\Delta\mathrm{Attn}$ (mean $\pm$ SEM). Negative values indicate reduced attention share to domain keywords.

Panel of Figure 3 — Figure 3: Instance-level correlations between $\Delta\mathrm{C1}$ (prompt Hoyer) and $\Delta\mathrm{C3}$ across models and contrasts. Positive values indicate that instances with larger decreases in Hoyer tend to also exhibit larger decreases in the corresponding instability measure (entropy or semantic variation).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\Delta\mathrm{Hoyer}$ Tag–Ctrl	$\Delta\mathrm{Hoyer}$ Instr–Ctrl	$\Delta\mathrm{Hoyer}$ Tag–Instr
Llama-3.1-8B-Instruct	$-0.015\pm 0.001$	$-0.016\pm 0.001$	$+0.001\pm 0.000$
Mistral-7B-Instruct-v0.2	$-0.016\pm 0.001$	$-0.017\pm 0.001$	$+0.000\pm 0.000$
Qwen3-8B	$-0.005\pm 0.000$	$-0.003\pm 0.000$	$-0.002\pm 0.000$

Appendix

Table 1: Early segment: mean

\pm

SEM of

\Delta\mathrm{Hoyer}

under three contrasts.

Model	$\Delta\mathrm{Hoyer}$ Tag–Ctrl	$\Delta\mathrm{Hoyer}$ Instr–Ctrl	$\Delta\mathrm{Hoyer}$ Tag–Instr
Llama-3.1-8B-Instruct	$-0.014\pm 0.001$	$-0.015\pm 0.001$	$+0.000\pm 0.000$
Mistral-7B-Instruct-v0.2	$-0.012\pm 0.001$	$-0.015\pm 0.001$	$+0.003\pm 0.000$
Qwen3-8B	$-0.007\pm 0.001$	$-0.009\pm 0.001$	$+0.002\pm 0.000$

Appendix

Table 2: Middle segment: mean

\pm

SEM of

\Delta\mathrm{Hoyer}

under three contrasts.

Model	$\Delta\mathrm{Hoyer}$ Tag–Ctrl	$\Delta\mathrm{Hoyer}$ Instr–Ctrl	$\Delta\mathrm{Hoyer}$ Tag–Instr
Llama-3.1-8B-Instruct	$-0.008\pm 0.000$	$-0.012\pm 0.000$	$+0.004\pm 0.000$
Mistral-7B-Instruct-v0.2	$-0.002\pm 0.000$	$-0.015\pm 0.000$	$+0.013\pm 0.000$
Qwen3-8B	$-0.001\pm 0.000$	$-0.007\pm 0.001$	$+0.006\pm 0.000$

Appendix

Table 3: Global segment: mean

\pm

SEM of instance-level paired differences in

\Delta\mathrm{Hoyer}

under three contrasts.

Model	$\Delta\mathrm{Attn}$ Tag–Ctrl (Mean $\pm$ SEM)	$\Delta\mathrm{Attn}$ Instr–Ctrl (Mean $\pm$ SEM)
Llama-3.1-8B-Instruct	$-0.0389\pm 0.0031$	$-0.0371\pm 0.0030$
Mistral-7B-Instruct-v0.2	$+0.0154\pm 0.0005$	$+0.0164\pm 0.0005$
Qwen3-8B	$-0.0409\pm 0.0032$	$-0.0396\pm 0.0031$

Appendix

Table 4: Mean

\pm

SEM of

\Delta\mathrm{Attn}

under Tag–Ctrl and Instr–Ctrl across three models (Prompt-last view).

Model	$\Delta\mathrm{Attn}$ Tag–Ctrl (Mean $\pm$ SEM)	$\Delta\mathrm{Attn}$ Instr–Ctrl (Mean $\pm$ SEM)
Llama-3.1-8B-Instruct	$-0.0592\pm 0.0045$	$-0.0599\pm 0.0046$
Mistral-7B-Instruct-v0.2	$+0.0158\pm 0.0004$	$+0.0175\pm 0.0005$
Qwen3-8B	$-0.0493\pm 0.0039$	$-0.0480\pm 0.0038$

Appendix

Table 5: Mean

\pm

SEM of

\Delta\mathrm{Attn}

under Tag–Ctrl and Instr–Ctrl across three models (First-gen view). Negative values indicate reduced domain-keyword attention share relative to control.

Model	$r(\Delta\mathrm{Attn},\,\Delta\mathrm{Entropy})$	$p$ -value
Llama-3.1-8B-Instruct	$0.117$	$1.3\times 10^{-10}$
Mistral-7B-Instruct-v0.2	$-0.072$	$7.8\times 10^{-5}$
Qwen3-8B	$0.229$	$5.4\times 10^{-37}$

Appendix

Table 6: Pearson correlation between

\Delta\mathrm{Attn}

and

\Delta\mathrm{Entropy}

(instance-level), with corresponding

p

-values.

Model	$r$ (Instr vs. Ctrl)	$r$ (Tag vs. Ctrl)	$r$ (Tag vs. Instr)
Llama-3.1-8B-Instruct	$0.141^{***}$	$0.013^{\mathrm{ns}}$	$0.105^{***}$
Mistral-7B-Instruct-v0.2	$0.191^{***}$	$0.043^{*}$	$0.150^{***}$
Qwen3-8B	$0.309^{***}$	$0.204^{***}$	$0.139^{***}$

Appendix

Table 7: Pearson correlations between C1 (

\Delta\mathrm{Hoyer}

) and C3-Entropy (

\Delta\mathrm{Entropy}

) under three contrasts. Superscripts indicate significance:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

, and ns not significant.

Model	Instr–Ctrl	Tag–Ctrl	Tag–Instr
Llama-3.1-8B-Instruct	$0.067^{***}$	$-0.027^{\mathrm{ns}}$	$0.037^{*}$
Mistral-7B-Instruct-v0.2	$0.142^{***}$	$0.128^{***}$	$0.177^{***}$
Qwen3-8B	$0.023^{\mathrm{ns}}$	$0.072^{***}$	$0.017^{\mathrm{ns}}$

Appendix

Table 8: Pearson

r

between C1 (

\Delta\mathrm{Hoyer}

) and C3-SemVar (

\Delta\mathrm{Var}

). Superscripts:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

\mathrm{ns}

not significant.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 349 papers

Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
May 22, 2026Md Nurul Absar SiddikyMixture-of-Experts Language Models Expert Routing
Department of Electrical and Computer Engineering University of Hawai’i at M¯anoa Honolulu, HI, USA
Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs
Expected Reward Prediction, with Applications to Model Routing
Mar 3, 2026Kenan Hasanaliyev, Silas Alberti, Jenny Hamer +5LLM Routing Reward Modeling
Stanford University · Inception Labs · Cognition Labs +2

May 22, 2026J/K move · Enter open · S save

Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts

Md Nurul Absar Siddiky

Department of Electrical and Computer Engineering University of Hawai’i at M¯anoa Honolulu, HI, USA

Mixture-of-Experts Language Models Expert Routing LLM Safety LLM Routing

Open paper AskPDF arXiv

Sparse mixture-of-experts (MoE) language models activate only a small subset of parameters for each token, making router behavior a central part of model computation. This paper studies routing behavior of Mixtral 8x7B-Instruct under benign and harmful prompts using two complementary signals: activation-based routing scores derived from expert selection frequencies and gradient-based scores derived from router-gate sensitivities. We analyze expert- and layer-level routing behavior and conduct expert-suppression interventions. The results show that activation-based expert usage is broad and long-tailed, whereas gradient-based importance is concentrated. At expert level, benign and harmful prompt groups remain close under both signals with modest separation. At layer level, activation-based routing is most selective around layers 8-15, while gradient-based importance is concentrated in final layers. Expert classification shows most experts are shared across benign and harmful prompts, though a limited subset shows clear group preference. Top-ranked expert sets show stronger benign-malicious overlap under gradient scores than activation scores, suggesting concentration on a common late-layer expert set. In intervention experiments, suppressing top five benign-dominant experts from activation scores reduces restricted responses from 24 to 14 over 100 prompts, while suppressing gradient-derived experts reduces them from 34 to 22 with fewer unintended reversals. Overall, safety-relevant routing in Mixtral is subtle, depth-dependent, and distributed rather than dominated by a fixed set of experts.

Abstract

LLM Routing

Explore similar work

Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts

Dynamic LLM Routers are Often Misguided

Expected Reward Prediction, with Applications to Model Routing