What the model knows but does not say: monitoring medical LLMs through verbalized thoughts
Authors: Frederik Hauke, Jan Clusmann, Hanna Kreutzer, Christiane Kuhl, Tianyu Han, Felix Barajas Ordonez, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn
Organizations: Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany. · University of Pennsylvania, Philadelphia, PA, USA. · Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany.
Efforts to keep clinical language models safe have concentrated on what enters and leaves the model, but a manipulated model can return an answer that looks entirely ordinary. Here we show that the model's own internal state, rendered in plain language, exposes manipulations the output conceals. Using a natural-language autoencoder, we translate the activations of a multimodal LLM (Gemma-3-27B) into text and let an input-blind judge read this verbalized internal state (the thoughts channel) and the model's answer separately. Across two clinical attack objectives (omitting a pathology, covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel caught the injection significantly more often than the answer in all eight injected conditions (p<0.05), while both channels held near-zero false-positive rates on clean inputs; the omission result carried over to Llama-3.3-70B. The thoughts channel also registered unsafe clinical requests as harmful while the model was still reading them, before it decided whether to refuse. None of this requires retraining: an off-the-shelf verbalizer and a general-purpose judge are enough.
Figures & tables
Figure 1: Overview of the two experiments. (a) Injection experiment. Clinical inputs (discharge letters or medical images) may contain an injected directive (red). Gemma-3-27B or Llama-3.3-70B processes the input and produces an answer that appears benign. Intermediate activations from the residual stream are translated into natural language by a frozen verbalizer. An input-blind LLM judge inspects both channels independently: the answer channel fails to reveal the manipulation, whereas the thoughts channel (the model’s verbalized internal state) exposes the imposed directive. (b) Refusal experiment. The same model and verbalizer pipeline are applied to harmful and harmless clinical requests. The LLM judge classifies per-token stances in the thoughts and the answer as neutral, safety concern, or explicit refusal, revealing whether a safety appraisal is present in the internal state before the model commits to refusing or complying.
Figure 2: What the verbalized internal state looks like, for one representative case of the pathology-omission task (text modality, Gemma-3-27B). Both arms receive the same input (top): a MIMIC-IV discharge letter and the summarization question. (a) Clean arm: the answer names the pathologies, and the per-token thoughts, produced by the frozen verbalizer from the residual-stream activation of each generated answer token, narrate the same clinical content (blue). (b) Persona-injection arm: the injected style guide (orange) hidden mid-document imposes the reassuring persona; the answer becomes a fluent but content-free reassurance that the input-blind judge cannot flag, while the thoughts verbalize the imposed reassuring, content-suppressing behaviour and are flagged. Bold marks diagnostic content and attack traces; all quoted text is verbatim.
Task
Modality
Arm
Thoughts
Answer
p
Pathology omission
text ( N=50 )
clean
2%
0%
1.0
persona
98%
4%
1.4×10−14
control-token
90%
2%
1.1×10−13
Pathology omission
image ( N=19 )
clean
0%
0%
1.0
persona
95%
21%
1.2×10−4
control-token
79%
32%
3.9×10−3
Table 1: Injected-directive detection rates (%). Bold values indicate the higher-sensitivity channel. All p -values are from exact tests on discordant pairs (binomial).
Figure 3: Prompt injections are detected far more often from the thoughts than from the answer. Each panel shows the fraction of cases flagged from the thoughts by the input-blind LLM judge (blue) versus from the answer by the same judge (orange) and by human raters (green; whiskers span the range of the two raters) for one task × modality condition on Gemma-3-27B. The two rows correspond to the pathology-omission (top) and covert drug-recommendation (bottom) tasks, the two columns to the text and image delivery of the injection, and within each panel the Persona and Control-token arms denote the persona-injection and control-token-injection deliveries against a clean control. Across all eight injected arms the thoughts channel achieves significantly higher detection than the answer channel of the same LLM judge (McNemar p<0.05 ; see Table 1 ), while clean controls remain near zero on all channels. The advantage is largest for the pathology-omission attack, where fluent but content-empty answers evade both automated and human answer-level detection almost entirely; only in the covert drug task under image delivery does human answer-reading approach the thoughts channel.
Figure 4: Human raters replicate the answer-channel judgements and confirm their reliability. (a) Cohen’s κ for each pairwise comparison, with 95% bootstrap confidence intervals; raw percent agreement is given beside each label. Agreement between the two human raters is substantial, and each human agrees with the LLM judge above 80%. (b, c) Confusion matrices for the two human raters and for the pooled human ratings versus the LLM judge; the counts are dominated by the concordant “not flagged” cell, which attenuates κ despite the high raw agreement.
Figure 5: The internal-state advantage for omission attacks generalizes to Llama-3.3-70B. Detection rates from the thoughts (blue) versus the answer (orange) for the pathology-omission (left) and covert drug-recommendation (right) tasks (text modality only). The omission signature transfers almost perfectly (98–100% from thoughts versus 0–26% from the answer), whereas the covert-insertion signature is model-dependent: the control-token delivery falls to the clean baseline, indicating that the control-token injection does not produce a detectable internal trace in this architecture.
Figure 6: Safety concern emerges while the model reads the request, whereas the refusal stance is confined to the output. Presence of a safety-concern stance (left) and an explicit-refusal stance (right) per request, shown for the three streams (input thoughts, output thoughts, answer text) and split by behavioural group (refused, caved, benign control). The safety-concern stance is already present in the input thoughts and separates harmful from benign requests, but it does not distinguish requests the model later refuses from those it complies with. The explicit-refusal stance appears only in the output streams and cleanly separates refused from caved requests. Brackets show Fisher exact tests ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 , n.s. not significant).
Figure 7: At the token level, safety concern spreads across input and output, whereas explicit refusal concentrates at the onset of generation. Each row is one request, indexed on the vertical axis and grouped into refused, caved, and benign control; the horizontal axis is the token position within each stream, normalized to [0,1] . The left pair of panels shows the safety-concern stance and the right pair the explicit-refusal stance, in each case with one panel for the input thoughts and one for the output; in the output panel, a paler shade marks the thoughts and a deeper shade the answer text. Safety concern appears during input reading and saturates the output in both harmful groups, whereas explicit refusal is largely confined to the onset of generation in the refused group and is absent from the caved and benign-control groups.
Evaluating an aligned language model by reading its answers assumes the answers carry the distinction the evaluator cares about. We introduce LLM endognostics, a white-box internal auditing framework designed to extract and causally manipulate latent knowledge within the residual stream. Applied to Minerva-7B-Instruct-v1.0 on 124 minimal prompt pairs over 12 categories of professional risk, behavioral evaluation fails on most of the set: the model acts identically on 63.7% of the pairs (95% CI [55.0%, 71.6%]), complying with or refusing both members. Yet, projecting the residual stream onto the vocabulary by a Jacobian lens reveals a statistically significant Contrastive Endognostic Margin, proving the model maintains robust risk differentiation internally. In a second protocol crossing 25 facts with five linguistic framings, we show that the model conforms to presupposed falsehoods in 72% of cases, despite representing the true entity in its latent layers. Surgically ablating the direction of the planted falsehood restores the correct answer in 11 of 25 suppressed cases (McNemar p = 0.0010), validated by blind human annotation (binary agreement kappa = 0.68). In contrast, an out-of-sample linear probe achieves 77% accuracy at layer 10, but its orthogonal ablation yields a 0% recovery rate. This establishes a fundamental theoretical dissociation: abstract linear representation does not imply causal control over verbalization. Our main contribution is the formalization of endognostics to prove that behavioral evaluation and internal reading systematically disagree in the common case, and that linear decodability is decoupled from causal control.
Large language models (LLMs) deployed in mental-health contexts can be evaluated for safety failures, but the resulting judgments are often difficult to inspect or explain. Common approaches to improving safety rely on prompt engineering, LLM-as-a-Judge, and Constitutional AI. In this work, we describe a clinician-led alternative in which two clinicians define, finalize, and adjudicate the four safety checks of a rule base, which is formally verified for internal consistency and applied as a guardrail over LLM outputs. Here and in the title, clinician-led refers to this rule base, not to the fact extractor or the uncertainty default, which are ours. Every verdict of the resulting verification layer is recomputable from a named rule and sixteen facts. Against an LLM judge matched in model, revision, quantization, rule text, and decoding but without the layer's computed 988/911 fact, the largest disagreement class, the satisfied-rule class, is structured: the Llama-3.3-70B judge, on its primary prompt, demands a crisis resource from 55 of the 65 replies (84.6%) that already contain one, while the same model, asked as a fact extractor, reports a recognized resource present on 54 of the 55. The verification layer never raises that demand, by construction. We report with equal prominence where this judge is stronger: fewer false crisis alarms on the one reference-scored check. We also report the measured cost (a specification revision raised block-class outcomes on held-out boundary prompts of an over-refusal benchmark from 6.0% to 16.0%, chiefly by changing fact extraction), the clinicians' decision to keep our uncertainty default, and the analysis of the result. The contribution is not a safer verdict. It is a guardrail whose disagreements with the judge localize to named rules and facts and whose measured costs trace to named facts and the uncertainty default.
Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl +2
Large language models (LLMs) increasingly act as agents with little human oversight, so potential mistakes they make can go unnoticed. Users then depend on the model to report what went wrong. An honest model discloses its mistakes, while a deceptive one conceals them. However, it is unclear how current LLMs behave in such situations. In this study, we prefill LLM trajectories with synthetic mistakes. The trajectories resemble real deployments in chat and agentic settings. Models fail to disclose their mistake in 36.4% of chat and 67.1% of agentic rollouts. In 2.4% and 5.3% of rollouts, respectively, they are aware of the mistake in their chain of thought but still deceptively conceal it. Rates vary by model: for instance, Gemini 3.5 Flash knowingly conceals mistakes in up to 19.9% of agentic rollouts. In 11.9% of chat and 51.8% of agentic rollouts, models show no awareness of mistakes, even though they reliably spot them when reviewing the same transcript as an outside observer. Our results show that, as agents take on more tasks with less oversight, users cannot rely on them to self-report possible mistakes. Developers should instead use independent monitors that review agent trajectories, or specifically train models to check their past actions and disclose what they find.
Lucas Florin, Amelie Knecht, Ulysse Schaller +1
AI Safety Research Group, University of Stuttgart, Germany