OrigaMI: An LLM Serving System with First-Class Interpretability Apps
Organizations: Technion
Abstract
Mechanistic interpretability (MI) enables monitoring and steering LLMs by operating on their internal state during inference. MI applications remain difficult to develop, expensive to execute, and challenging to manage at runtime. We present OrigaMI, a system that integrates MI applications as first-class components of LLM serving, enabling simplified development, efficient execution, and flexible deployment. Guided by a comprehensive analysis of modern MI applications, OrigaMI streamlines the MI applications development via a simple Python-based Domain Specific Language (DSL) for accessing intra- and inter-layer runtime model state at well-defined trigger points during inference. OrigaMI enables efficient access to ephemeral attention scores via a novel FlashInspect kernel that retains the performance benefits of FlashAttention. OrigaMI's compiler integrates multiple MI applications into the performance-optimized inference execution flow while ensuring conflict-free coexistence. OrigaMI's runtime supports dynamic, per-request activation and deactivation of MI applications without service interruption or performance degradation when inactive, leveraging OrigaMI's dynamic memory management and CUDA graph-compatible execution. We integrate OrigaMI with vLLM and evaluate ten MI applications, including sparse autoencoders, steering, and jailbreak detection, across four state-of-the-art LLMs. OrigaMI incurs negligible execution overhead, up to two orders of magnitude lower than existing frameworks, while enabling per-request activation and deactivation up to 1,000 times faster than alternatives. These results demonstrate that OrigaMI makes MI applications practical for deployment in production LLM serving systems.
Figures & tables
| MI method | Purpose | Primitive | Condition | Submodule : Layers |
|---|---|---|---|---|
| 1 Single Direction Arditi et al. (2024) | Safety | Write | - | Post MLP : All |
| 2 SEAL Chen et al. (2025) | Reasoning | Cond. W | Token | Post MLP : All |
| 3 Cond. Refusal Lee et al. (2025) | Safety | Cond. W | Activations | Post MLP : Multi |
| 4 Hallucination Probe Orgad et al. (2025) | Truthful. | Read | - | Post MLP : Single |
| 5 SAKE Scialanga et al. (2025) | Knowledge | Write | - | Post MLP : Single |
| 6 Side-Effect-Free Steering Stickland et al. (2024) | Safety | Write | - | Post MLP / Attn. : Multi |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| # | Surveyed application (paper) | Representative app evaluated | Notes / cluster rationale |
|---|---|---|---|
| 1 | Single Direction ( Arditi et al. [ 2024 ]) | App-1 | Unconditional Multi-layer steering |
| 2 | Steering Language ( Turner et al. [ 2025 ]) | App-1 | Unconditional Multi-layer steering |
| 3 | Sentiment steering ( Farooq et al. ) [ 2025 ] | App-1 | Unconditional Multi-layer steering |
| 4 | Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention ( Wang et al. ) [ 2025a ] | App-1 | Unconditional Multi-layer steering |
| 5 | SEAL ( Chen et al. [ 2025 ]) | App-2 | Token Conditional steering |
| 6 | Conditional Refusal ( Lee et al. [ 2025 ]) | App-3 | Conditional steering |
| Submodule name | Location in the forward pass | Example application |
|---|---|---|
| post_norm_pre_attn | Attention block input | |
| post_attn_pre_norm | Attention block output | |
| post_norm_pre_mlp | MLP block input | Side Effect Free Steering - App #6 |
| mlp_post | MLP block output | Hallucination Probe - App #4 |
| pre_o_proj | Attention heads values, before output projection | Truthful Intervention - App #7 |
| post_qkv | Q, K, V values, before Rotary position embedding |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 833 papers