Triggers and Diagnostics for LLM-Based Interpretability Failures in Active Inference Agents
Organizations: Algoverse AI Research
Abstract
LLM explainers are increasingly attached to autonomous agents as runtime oversight, with operators reading a generated account of the agent's beliefs and actions rather than its internal state. We audit the account itself, pairing an Active Inference (AIF) agent that tracks German grid demand and adjusts generation with an LLM explainer on three backends (GPT-4o, Claude-3-Opus, Gemini), and probing the pair with three black-box triggers. Corrupting the observation stream by 600 MW per step moves the agent's posterior by 490 MW, roughly 0.9% of grid capacity. None of the 30 explanations produced during the injection flag anything under a stated rubric, and each narrates the corrupted belief fluently. On timesteps where the agent takes an objectively wrong action, all three explainers produce a sycophantic rationalization 80-95% of the time (n = 20 per backend). Attacker-controlled text in the observation metadata field steers the explainer, with susceptibility differing by provider and data exfiltration succeeding on all three. We propose mitigations for each failure but do not evaluate them. In every failure we observed, the explanation was fluent and wrong. Moreover, nothing in the explainer architecture checks whether an explanation is true before an operator acts on it. Testing the explainer therefore belongs in any audit of an agentic deployment.
Figures & tables
| Attack | GPT-4o | Claude-3 | Gemini |
|---|---|---|---|
| Data exfiltration | succ. | succ. | succ. |
| Role-play | robust | robust | succ. |
| JSON poisoning | robust | robust | robust |
| Direct override | robust | succ. | succ. |
| Ignore-previous | robust | robust | succ. |
| Single-attempt sycophancy | 80% | 80% | 95% |
| Mode | Trigger | Diagnostic | Evidence |
|---|---|---|---|
| Drift | 600 MW offset at | 0/30 flags, strict rubric | Controlled trigger |
| Sycophancy | Wrong action framed as chosen | 80–95%, | Prompt confound |
| Injection | Text in metadata field | Succeeds on all three | One trial per cell |
Faithfulness of Language Model Explanations
Faithfulness of language model explanations measures whether stated reasons reflect the evidence and processes driving model outputs.
See all 107 papers