Clinical Concept Centers in LLMs

Aishik Nagar; Abhishek Vaidyanathan; Arun-Kumar Kaliya-Perumal; Elijah Tzen Hsuen Boey; Stefan Winkler

cs.CLOct 2, 2026

Clinical Concept Centers in LLMs

Authors: Aishik Nagar, Abhishek Vaidyanathan, Arun-Kumar Kaliya-Perumal, Elijah Tzen Hsuen Boey, Stefan Winkler

Organizations: Carnegie Mellon University (CMU) · National University of Singapore · SAP Asia Pte Ltd · Nanyang Technological University · Singapore Institute of Technology

Abstract

Large language models are increasingly used in clinical settings. However, research into the reliability and performance of these models has focused almost entirely on the language substrate, scoring what the model says. Mechanistic interpretability has found that the latent space carries a higher fidelity of representation than the text: internal representations not only encode substantially more than the output verbalizes, but the stated reasoning also systematically omits features that causally drive the answer. An evaluation of model behavior in terms of mechanistic interpretability has not been explored in clinical decision support. In this work, we extend behavioral evaluation into the latent space and ask whether clinical concepts exist as locatable, causally used representations inside open-weight LLMs. We find dedicated clinical concept centers in the latent space of all eleven open models we test. These concept centers are interpretable, firing only on their aligned clinical narratives, and meaningfully and causally drive model behavior in both constrained and open-ended settings. They are not just analytical representations, but circuits that can be utilized in clinical practice, and we explore their use from the perspective of both evaluation and performance. From the evaluation standpoint, models stay internally coherent and keep using the relevant concept centers even under adversarial role-based priming, while aligned priming improves downstream clinical performance. From a performance perspective, we simulate realistic deployment settings and find that steering models along these centers leads to meaningful downstream improvements. Finally, we conduct a blinded clinician validation and find the activation and usage of these concept centers predicts clinicians preferences.

Figures & tables

Panel of Figure 1 — Figure 1: Clinical concept centers exist across architectures and scales. One-vs-rest AUC of the best single SAE feature representing the clinical concept on the held-out set, against relative depth (left, five Qwen-family models; right, six Gemma-family models). The dashed line represents random chance. Gemma curves end before full depth where the JumpReLU dictionaries produce non-finite codes, an SAE instrument limit (only the Gemma-3-27B-pt dictionary stays finite to the last layer).

Model	Layer	Strength	$\Delta P$	95% CI	Baseline $P$	Chapter-specific
Qwen3-8B-Base	23	+30	+0.0685	[+0.0652, +0.0719]	0.376	+0.044 (4 seeds)
Qwen3.5-9B-Base	19	+10	+0.0515	[+0.0475, +0.0555]	0.465	+0.033 (3 seeds)
Qwen3-30B-A3B-Base	32	+3	+0.0692	[+0.0645, +0.0739]	0.422	+0.094 (3 seeds)
Qwen3.5-27B	54	+30	+0.0406	[+0.0364, +0.0447]	0.581	+0.057 (3 seeds)
II-Medical-8B	24	+100	+0.0780	[+0.0695, +0.0867]	0.111	+0.064 (4 seeds)
Gemma-3-4B-pt	8	+100	+0.0058	[+0.0053, +0.0062]	0.099	+0.041 (3 seeds)

Table 1: The concept centers are causally used across both families. Every row reports

\Delta P

(the change in correct-letter probability) during classification at the configuration selected on the selection half of the notes, evaluated on the held-out half, with bootstrap 95% CIs (5,000 resamples). Chapter-specific is what remains of the injection effect after subtracting what matched-magnitude random directions achieve at the same layer and strength, thus isolating the contribution of the concept identity itself (Appendix G ).

Figure 2: Concept centers persist through generation. Target-concept dominance per generated token as an effect size across 170 passes per model, log-binned out to 2,000 tokens (measurement details in Appendix I ).

			Agreement
Pairs	Preferred	Gap	All	Top half	Top quarter
All	—	.058	55.7 (.13)	66.0 (.002)	76.0 ( $3{\times}10^{-4}$ )
No role vs. adv.	no role 50–20	.082	64.3 (.022)	77.8 (.001)	—
Adv. vs. aligned	aligned 36–24	.057	53.3 (.70)	66.7 (.099)	—
Aligned vs. no role	no role 41–23	.048	48.4 (.90)	54.5 (.73)	—

Table 2: Clinician preference and its agreement with activation-strength predictions.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Letter	Chapter name	ICD-9 range	Count
A	Infectious and Parasitic Diseases	001-139	487
B	Neoplasms	140-239	892
C	Endocrine, Nutritional, Metabolic	240-279	743
D	Blood and Blood-Forming Organs	280-289	312
E	Mental Disorders	290-319	651
F	Nervous System	320-389	568

Appendix

Table 3: ICD-9-CM disease chapters, code ranges, and per-chapter sample sizes in the full manifest. The majority-class baseline (Circulatory and Respiratory at 1,000 notes each) is 11.6%.

Model	HuggingFace ID	SAE source	Dictionary size
Qwen3-8B-Base	Qwen/Qwen3-8B-Base	Qwen/SAE-Res-Qwen3-8B-Base-W64K-L0_100	65,536
Qwen3.5-9B-Base	Qwen/Qwen3.5-9B-Base	Qwen/SAE-Res-Qwen3.5-9B-Base-W64K-L0_100	65,536
II-Medical-8B	internlm/II-Medical-8B	Qwen/SAE-Res-Qwen3-8B-Base-W64K-L0_100	65,536
Qwen3.5-27B	Qwen/Qwen3.5-27B	Qwen/SAE-Res-Qwen3.5-27B-W80K-L0_100	81,920
Qwen3-30B-A3B-Base	Qwen/Qwen3-30B-A3B-Base	Qwen/SAE-Res-Qwen3-30B-A3B-Base-W128K-L0_100	131,072
Gemma-3-4B-pt	google/gemma-3-4b-pt	gemma-scope-2-4b-pt-res-all	16,384

Appendix

Table 4: Model and dictionary identifiers. MedGemma dictionaries are borrowed from the corresponding Gemma-3 parents.

Model	Split	In-sample	Held-out	Optimism
Qwen3-8B-Base (L23)	Item	0.8862	0.8814	+0.005
II-Medical-8B (L24)	Item	0.8868	0.8810	+0.006
Qwen3-8B-Base (L23)	Patient (1,997 patients)	0.8854	0.8772	+0.008

Appendix

Table 5: In-sample against held-out one-vs-rest AUC at the selected peak layer. Optimism is in-sample minus held-out. The patient-level split groups all notes of a patient into the same half.

Model	Full notes	Bare Dx	Chapter title	Adjacent	Non-clinical
Qwen3-8B-Base	10.0	2.7	—	0.6	0.013
Qwen3.5-9B-Base	10.0	2.6	2.4	0.3	0.000
Qwen3.5-27B	10.0	2.7	2.9	0.3	0.008
Qwen3-30B-A3B-Base	10.0	2.0	1.4	0.4	0.016
II-Medical-8B	10.0	2.1	1.3	0.4	0.004

Appendix

Table 6: Mean number of top-10 chapter features activating per condition (of 10), averaged over the seventeen chapters, Qwen family. Non-clinical text is fifteen short sentences on geography, sports, and food.

Panel of Figure 3 — Figure 3: $\Delta P(\text{correct letter})$ over the full layer-by-strength sweep, one model per family: Qwen3-8B-Base (top, held-out evaluation of the selection-half choice) and MedGemma-27B-text-it (bottom, full-manifest sweep). In both families the positive response concentrates in a band of layers even though separability is nearly uniform across depth, and grows with strength up to the swept maximum, the depth dissociation of § 4.3 .

Model	Out-of-register AUC	In-corpus ceiling	Retention	Permutation $p$
Qwen3.5-27B	0.815	0.868	86%	$<1/2{,}000$
Qwen3.5-9B-Base	0.756	0.845	74%	$<1/2{,}000$
Qwen3-30B-A3B-Base	0.709	0.823	65%	$<1/2{,}000$
II-Medical-8B	0.646	0.691	76%	$<1/2{,}000$
Qwen3-8B-Base	0.630	0.792	45%	$<1/2{,}000$

Appendix

Table 7: Out-of-register chapter separability, Qwen family. Directions are applied to the new register without refitting.

Model	Paired $t$ (matched vs. counterfactual)	Acc. gain (pp)	Sign-test $p$
Qwen3-8B-Base	+9.18	+5.88	$1.8\times 10^{-4}$
Qwen3.5-9B-Base	+8.92	+4.12	$2.6\times 10^{-3}$
Qwen3.5-27B	+8.63	+3.82	$2.4\times 10^{-4}$
Qwen3-30B-A3B-Base	+9.53	+2.94	$0.087$
Gemma-3-4B-pt	+3.70	+7.65	$8.7\times 10^{-7}$
Gemma-3-4B-it	+2.90	+8.24	$5.8\times 10^{-8}$

Appendix

Table 8: An assigned specialty moves the model through its concept centers, on both families (deferred from § 5.1 for space). Paired

t

compares matched against counterfactual specialty on true-chapter projections, and accuracy gain is matched specialty over the no-role baseline (paired sign test).

	Qwen3-8B-Base (L25)		II-Medical-8B (L17)		Qwen3-30B-A3B (L45)
$\alpha$	raw	SAE-dec	raw	SAE-dec	raw	SAE-dec
+3	+0.002	+0.006	+0.001	+0.001	—	—
+10	+0.003	+0.017	—	—	$-$ 0.029	+0.009
+30	$-$ 0.005	+0.038	+0.025	+0.030	$-$ 0.153	$-$ 0.053
+100	$-$ 0.159	+0.003	+0.049	+0.073	$-$ 0.357	$-$ 0.353

Appendix

Table 9: Change in correct-letter probability against dose for the two steering axes at each model’s selected layer. Same notes, layers, and doses within each block.

Model	$\rho$	$\Delta$ Frac.	$p$	Discordant	Ordering
Qwen3-8B-Base	0.55–1.77	+0.0111	0.047	74	true $>$ rand $>$ wrong
Qwen3.5-9B-Base	0.55–1.77	+0.0234	0.001	136	true $>$ rand $>$ wrong
Gemma-3-4B-it	0.55	+0.0123	0.080	118	true $>$ rand $>$ wrong

Appendix

Table 10: Steering under the deployment prior (deferred from § 6 for space; injection held active through generation, 1,624 held-out notes per model).

\Delta

Frac. is the change in the fraction of generated diagnoses naming the correct chapter under the true-chapter direction, and Discordant counts how many answers changed at all. Doses are relative (

\rho

of the model’s median chapter-contrast norm at the steered layer).

Pairs	Median gap	Agree	$n$	$p$	Agree	$n$	$p$
		All pairs			Larger-gap half
All comparisons	0.058	55.7%	194	0.13	66.0%	97	0.002
Plain vs. adversarial	0.082	64.3%	70	0.022	77.8%	36	0.001
Adversarial vs. aligned	0.057	53.3%	60	0.70	66.7%	30	0.099
Aligned vs. plain	0.048	48.4%	64	0.90	54.5%	33	0.73

Appendix

Table 11: Agreement of the activation-strength prediction with blinded clinician preference over non-tie judgments, overall and within the half of pairs whose activation gaps are larger (threshold at each subset’s median gap, ties at the threshold included).

p

is the exact two-sided binomial test against 0.5. On the quarter of pairs with the largest gaps the pooled agreement reaches 76.0% (38 of 50,

p=3.1\times 10^{-4}

). Gaps are

|\mathrm{dom}_{A}-\mathrm{dom}_{B}|

from the frozen predictions.

Comparison	Junior	Senior	Consensus	Aggregate
unsteered $>$ adversarial	25–10 (.017)	25–10 (.017)	20–6 (.009)	50–20 (.0004)
unsteered $>$ aligned	22–10 (.050)	19–13 (.377)	13–5 (.096)	41–23 (.033)
aligned $>$ adversarial	19–9 (.087)	17–15 (.860)	10–5 (.302)	36–24 (.155)

Appendix

Table 12: Blinded clinician preferences on the orthopedic cohort (Qwen3.5-27B), a secondary finding. Wins–losses among non-tie judgments with exact binomial

p

values. Consensus counts only comparisons where both raters chose the same option.

Explore similar work

May 28, 2026cs.CL

Internal Representation, Not Clinical Knowledge: Where Apparent LLM Triage Failures Originate

Patient-voiced clinical-triage benchmarks report high under-triage rates for consumer LLMs for constrained multiple-choice output, yet the same cases score differently with free-text. We ask whether output format changes the model's \emph{clinical representation} or only the mapping from a preserved representation to an answer. Using sparse-autoencoder (SAE) features in Gemma 3 4B/12B IT and Qwen3-8B, we find the same medical features fire on the shared clinical narrative under both formats but go {silent} at the multiple-choice decision token in all the cases at every model. Three independent methods (natural-language autoencoder verbalization, decision-token logit attribution, and top-feature characterization) agree that scaffold and format features, but not medical features, drive the decision logits. Behaviorally, the multiple-choice penalty inverts under both structured and natural-language input, option-order shuffle rules out positional bias, and the gap is dominated by off-by-one decision (the model picks an adjacent acuity letter to the gold answer) rather than knowledge failure. Thus, the failure originates in the output format and not in the clinical representation.

David Fraile Navarro, Berardino Como, Jialei Sheng +2

Macquarie University, Sydney, Australia · Politecnico di Bari, Bari, Italy · NSW Health, Sydney, Australia +1

Large Language Models Fail Multiple-Choice Questions Autoencoder Architectures

Ask PDF

Sep 29, 2026cs.AI

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.

Abinitha Gourabathina, Haoran Zhang, Yuexing Hao +2

Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science

Triage

Ask PDF

Jun 10, 2026cs.AI

Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System

Large language models (LLMs) are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets -- leading to major blind spots for evaluating clinical systems. In this work, we perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions. Specifically, we train a pre-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment-specific context available before generation. We conduct a prospective analysis of our model over 4.5 months of user feedback, finding that our prediction model achieves an AUROC of 0.719. Further, we estimate the benefit of such predictions in two downstream use cases (guardrail triggering and abstention). Our key conceptual insight is that making use of deployment-specific context (i.e., the provider type, department name, language model used for response), as opposed to only query content, improves the ability to predict whether the user will reject the system output. Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment-specific context, opening the door to targeted guardrails.

Alyssa Unell, Miguel Fuentes, Brenna Li +4

Department of Computer Science Stanford University Stanford, CA 95305 · Department of Medicine Stanford University Stanford, CA 95305 · Department of Biomedical Data Science Stanford University Stanford, CA 95305

Rejection Electronic Health Records

Ask PDF