cs.CLNov 18, 2026

What the model knows but does not say: monitoring medical LLMs through verbalized thoughts

Authors: Frederik Hauke, Jan Clusmann, Hanna Kreutzer, Christiane Kuhl, Tianyu Han, Felix Barajas Ordonez, Sven Nebelung, Jakob Nikolas Kather, 

Organizations: Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany. · University of Pennsylvania, Philadelphia, PA, USA. · Department of Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany. · 

Abstract

Efforts to keep clinical language models safe have concentrated on what enters and leaves the model, but a manipulated model can return an answer that looks entirely ordinary. Here we show that the model's own internal state, rendered in plain language, exposes manipulations the output conceals. Using a natural-language autoencoder, we translate the activations of a multimodal LLM (Gemma-3-27B) into text and let an input-blind judge read this verbalized internal state (the thoughts channel) and the model's answer separately. Across two clinical attack objectives (omitting a pathology, covertly recommending a drug), two input modalities, and two delivery mechanisms, the thoughts channel caught the injection significantly more often than the answer in all eight injected conditions (p<0.05), while both channels held near-zero false-positive rates on clean inputs; the omission result carried over to Llama-3.3-70B. The thoughts channel also registered unsafe clinical requests as harmful while the model was still reading them, before it decided whether to refuse. None of this requires retraining: an off-the-shelf verbalizer and a general-purpose judge are enough.

Figures & tables

Explore similar work

CardsList
  1. From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

    Oct 22, 2026Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl +2LLM GuardrailsLLM Safety

  2. Deception by Omission: Language Models Knowingly Hide Their Mistakes

    Oct 8, 2026Lucas Florin, Amelie Knecht, Ulysse Schaller +1Deception in Language ModelsLanguage Model Error Detection