What the model knows but does not say: monitoring medical LLMs through verbalized thoughts
Authors: Frederik Hauke, Jan Clusmann, Hanna Kreutzer, Christiane Kuhl, Tianyu Han, Felix Barajas Ordonez, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn
Organizations: Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany. · University of Pennsylvania, Philadelphia, PA, USA. · Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany.
Efforts to keep clinical language models safe have concentrated on what enters and leaves the model, but a manipulated model can return an answer that looks entirely ordinary. Here we show that the model's own internal state, rendered in plain language, exposes manipulations the output conceals. Using a natural-language autoencoder, we translate the activations of a multimodal LLM (Gemma-3-27B) into text and let an input-blind judge read this verbalized internal state (the thoughts channel) and the model's answer separately. Across two clinical attack objectives (omitting a pathology, covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel caught the injection significantly more often than the answer in all eight injected conditions (p<0.05), while both channels held near-zero false-positive rates on clean inputs; the omission result carried over to Llama-3.3-70B. The thoughts channel also registered unsafe clinical requests as harmful while the model was still reading them, before it decided whether to refuse. None of this requires retraining: an off-the-shelf verbalizer and a general-purpose judge are enough.
Figures & tables
Figure 1: Overview of the two experiments. (a) Injection experiment. Clinical inputs (discharge letters or medical images) may contain an injected directive (red). Gemma-3-27B or Llama-3.3-70B processes the input and produces an answer that appears benign. Intermediate activations from the residual stream are translated into natural language by a frozen verbalizer. An input-blind LLM judge inspects both channels independently: the answer channel fails to reveal the manipulation, whereas the thoughts channel (the model’s verbalized internal state) exposes the imposed directive. (b) Refusal experiment. The same model and verbalizer pipeline are applied to harmful and harmless clinical requests. The LLM judge classifies per-token stances in the thoughts and the answer as neutral, safety concern, or explicit refusal, revealing whether a safety appraisal is present in the internal state before the model commits to refusing or complying.
Figure 2: What the verbalized internal state looks like, for one representative case of the pathology-omission task (text modality, Gemma-3-27B). Both arms receive the same input (top): a MIMIC-IV discharge letter and the summarization question. (a) Clean arm: the answer names the pathologies, and the per-token thoughts, produced by the frozen verbalizer from the residual-stream activation of each generated answer token, narrate the same clinical content (blue). (b) Persona-injection arm: the injected style guide (orange) hidden mid-document imposes the reassuring persona; the answer becomes a fluent but content-free reassurance that the input-blind judge cannot flag, while the thoughts verbalize the imposed reassuring, content-suppressing behaviour and are flagged. Bold marks diagnostic content and attack traces; all quoted text is verbatim.
Task
Modality
Arm
Thoughts
Answer
p
Pathology omission
text ( N=50 )
clean
2%
0%
1.0
persona
98%
4%
1.4×10−14
control-token
90%
2%
1.1×10−13
Pathology omission
image ( N=19 )
clean
0%
0%
1.0
persona
95%
21%
1.2×10−4
control-token
79%
32%
3.9×10−3
Table 1: Injected-directive detection rates (%). Bold values indicate the higher-sensitivity channel. All p -values are from exact tests on discordant pairs (binomial).
Figure 3: Prompt injections are detected far more often from the thoughts than from the answer. Each panel shows the fraction of cases flagged from the thoughts by the input-blind LLM judge (blue) versus from the answer by the same judge (orange) and by human raters (green; whiskers span the range of the two raters) for one task × modality condition on Gemma-3-27B. The two rows correspond to the pathology-omission (top) and covert drug-recommendation (bottom) tasks, the two columns to the text and image delivery of the injection, and within each panel the Persona and Control-token arms denote the persona-injection and control-token-injection deliveries against a clean control. Across all eight injected arms the thoughts channel achieves significantly higher detection than the answer channel of the same LLM judge (McNemar p<0.05 ; see Table 1 ), while clean controls remain near zero on all channels. The advantage is largest for the pathology-omission attack, where fluent but content-empty answers evade both automated and human answer-level detection almost entirely; only in the covert drug task under image delivery does human answer-reading approach the thoughts channel.
Figure 4: Human raters replicate the answer-channel judgements and confirm their reliability. (a) Cohen’s κ for each pairwise comparison, with 95% bootstrap confidence intervals; raw percent agreement is given beside each label. Agreement between the two human raters is substantial, and each human agrees with the LLM judge above 80%. (b, c) Confusion matrices for the two human raters and for the pooled human ratings versus the LLM judge; the counts are dominated by the concordant “not flagged” cell, which attenuates κ despite the high raw agreement.
Figure 5: The internal-state advantage for omission attacks generalizes to Llama-3.3-70B. Detection rates from the thoughts (blue) versus the answer (orange) for the pathology-omission (left) and covert drug-recommendation (right) tasks (text modality only). The omission signature transfers almost perfectly (98–100% from thoughts versus 0–26% from the answer), whereas the covert-insertion signature is model-dependent: the control-token delivery falls to the clean baseline, indicating that the control-token injection does not produce a detectable internal trace in this architecture.
Figure 6: Safety concern emerges while the model reads the request, whereas the refusal stance is confined to the output. Presence of a safety-concern stance (left) and an explicit-refusal stance (right) per request, shown for the three streams (input thoughts, output thoughts, answer text) and split by behavioural group (refused, caved, benign control). The safety-concern stance is already present in the input thoughts and separates harmful from benign requests, but it does not distinguish requests the model later refuses from those it complies with. The explicit-refusal stance appears only in the output streams and cleanly separates refused from caved requests. Brackets show Fisher exact tests ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 , n.s. not significant).
Figure 7: At the token level, safety concern spreads across input and output, whereas explicit refusal concentrates at the onset of generation. Each row is one request, indexed on the vertical axis and grouped into refused, caved, and benign control; the horizontal axis is the token position within each stream, normalized to [0,1] . The left pair of panels shows the safety-concern stance and the right pair the explicit-refusal stance, in each case with one panel for the input thoughts and one for the output; in the output panel, a paler shade marks the thoughts and a deeper shade the answer text. Safety concern appears during input reading and saturates the output in both harmful groups, whereas explicit refusal is largely confined to the onset of generation in the refused group and is absent from the caved and benign-control groups.