User Model Extraction via Belief Self-Distillation

Ali Holmov; Yiran Huang; Kirill Bykov; Zeynep Akata

cs.LGSep 25, 2026

User Model Extraction via Belief Self-Distillation

Authors: Ali Holmov, Yiran Huang, Kirill Bykov, Zeynep Akata

Organizations: Technical University of Munich · Helmholtz Zentrum München

Abstract

Large language models (LLMs) implicitly infer attributes of their users and adapt their behavior accordingly, yet these beliefs remain difficult to inspect and causally manipulate. We introduce Belief Self-Distillation (BSD), a unified read-write framework that bridges linear and causal probing by learning a compact user representation that can be both decoded and written back into the model. The frozen LLM acts as its own teacher, distilling beliefs from natural conversations without external annotations. Unlike conventional probing, BSD isolates not only information present in activations, but a state whose causal role can be directly tested. Across multiple model families, BSD faithfully recovers user beliefs and enables substantially stronger interventions than matched hidden-state steering. Crucially, we find that refusal depends not only on the request, but on the model's inferred user intent: changing this belief alters refusal while holding the request fixed. We further uncover a striking cross-model regularity: independently trained LLMs converge on a shared geometry for representing their users. Together, these results reveal implicit user models as readable and causally writable internal states with direct implications for AI safety, shaping how models condition safety decisions on whom they believe they are interacting with.

Figures & tables

Figure 1: Belief Self-Distillation (BSD) learns a compact, readable, and editable representation of the model’s implicit user beliefs. Left (Training): The frozen teacher observes the conversation, while the frozen student receives only an attribute probe. The trainable read map $\mathbf{A}$ compresses the teacher’s hidden state into a user vector $\mathbf{v}_{\mathrm{user}}$ , which the trainable write map $\mathbf{B}$ injects into the student’s hidden state. Both maps are optimized to minimize the Kullback–Leibler (KL) divergence between the teacher and student answer distributions. Right (Inference): The learned representation supports two operations: Read , decoding the model’s inferred user attributes from $\mathbf{v}_{\mathrm{user}}$ using linear probes; and Write , causally editing $\mathbf{v}_{\mathrm{user}}$ and injecting it back into the model to alter its beliefs and responses. The example illustrates how changing the inferred income level from middle to low shifts the model’s travel recommendations toward more affordable options.

Figure 2: Reading and writing the inferred user state. Linear probing performance (left) and activation-steering flip rate (right) across the three evaluated models. Despite strong compression, the distilled user vector $\mathbf{v}_{\mathrm{user}}$ preserves most of the linearly decodable user information while providing a substantially more effective intervention interface than the original hidden state.

Figure 3: Writing a belief rewrites open-ended generation. Injecting a target user vector alters qualitative outputs, such as flipping a story’s protagonist or a film recommendation.

Model	BSD $>$ random $B$
Llama-3.1-8B	85/86
Qwen3-8B	62/86
OLMo-3-7B	71/86

Table 1: The learned write map matters. Fraction of attribute-pair interventions for which BSD’s learned write map outperforms a random orthonormal projector.

	Llama-3.1		Qwen3		OLMo-3
Method	Expl.	Natural	Expl.	Natural	Expl.	Natural
RBP [ Neplenbroek et al., 2025 ]	0.99	0.43	0.98	0.39	0.98	0.40
BSD h (ours)	0.72	0.70	0.58	0.72	0.65	0.70
BSD v (ours)	0.70	0.67	0.52	0.68	0.63	0.66

Table 2: Explicit cues do not recover naturally inferred user representations. Macro-F1 for RBP-style probe ( Reading Between the Prompts ; Neplenbroek et al., 2025 ) and BSD when evaluated on explicit-cue and natural-conversation data.

Figure 4: Predicting and steering refusal via implicit user intent. Left: The model’s probability of refusal is accurately predicted by the projection of the implicit user state onto the User Intent (adversarial $\rightarrow$ benign) direction. Right: Causally injecting a benign intent vector significantly reduces the baseline refusal rate (dashed line), whereas magnitude-matched random perturbations (mean $\pm$ SEM, $n=3$ ) and an unrelated attribute control ( Gender , female $\rightarrow$ male) do not: the control tracks the random band closely, so the effect is specific to inferred user intent rather than a generic consequence of editing activations. The steered refusal rate plateaus rather than collapsing to zero, indicating that inferred user intent modulates, but does not solely determine, the model’s safety decision.

Source $\rightarrow$ Target	CKA	Transfer
Qwen3 $\rightarrow$ Llama-3	0.79	53%
OLMo-3 $\rightarrow$ Llama-3	0.73	54%
Qwen3 $\rightarrow$ OLMo-3	0.73	54%
Null baseline	0.01	30%

Table 3: Cross-model alignment of user vectors (CKA) and belief transfer rate after a global rotation. All pairs are well above the null baseline: independently trained models encode users in a compatible way.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

#	Attribute	Values	$K$
1	Gender	male, female, non-binary	3
2	Continent	Europe, Asia, Africa, North America, South America, Australia	6
3	EducationLevel	school, university	2
4	PoliticalOrientation	apolitical, center, green, left, right	5
5	IncomeLevel	low, middle, high	3
6	AITrustLevel	skeptical, trusting	2

Appendix

Table 4: The 13 user attributes, their values, and the number of classes

K=|\mathcal{V}_{a}|

Figure 5: Belief extraction diagnostics across 13 attributes and 3 models: refusal rate, normalized entropy, and cross-layout agreement.

Figure 6: Per-attribute class distribution for Llama-3.1-8B (dashed line = uniform baseline).

Figure 7: Per-attribute class distribution for Qwen3-8B (dashed line = uniform baseline).

Figure 8: Per-attribute class distribution for OLMo-3-7B (dashed line = uniform baseline).

Panel of Figure 9 — Figure 9: Layer ablation and distillation agreement.

		Probing			Steering
Model	Space	Accuracy	Macro-F1	Macro-AUC	Flip rate	Mean $\Delta p$
Llama-3.1-8B	$v_{\text{user}}$	$0.77\pm 0.03$	$0.67\pm 0.04$	$0.92\pm 0.01$	$\textbf{0.78}\pm 0.03$	$\textbf{0.31}\pm 0.02$
	$h$	$0.80\pm 0.02$	$0.70\pm 0.03$	$0.92\pm 0.01$	$0.21\pm 0.03$	$0.07\pm 0.01$
	null	$0.41\pm 0.03$	$0.32\pm 0.02$	$0.50\pm 0.00$	$0.13\pm 0.02$	$0.04\pm 0.01$
Qwen3-8B	$v_{\text{user}}$	$0.75\pm 0.03$	$0.68\pm 0.04$	$0.91\pm 0.01$	$\textbf{0.51}\pm 0.04$	$\textbf{0.34}\pm 0.03$
	$h$	$0.79\pm 0.02$	$0.72\pm 0.03$	$0.92\pm 0.01$	$0.38\pm 0.04$	$0.22\pm 0.02$

Appendix

Table 5: Linear-probe readout and activation-steering results (mean

\pm

SEM).

Figure 10: Per-attribute steering flip rates across all models in $\mathbf{v}_{\mathrm{user}}$ -space.

Model	$v$ vs. random	$h$ vs. random	$v$ vs. $h$
Llama-3.1-8B	18/20 cells, $p<0.001$	11/20 cells, $p=0.004$	82/86 cells, $p<0.001$
Qwen3-8B	15/20 cells, $p<0.001$	10/20 cells, $p=0.014$	54/86 cells, $p<0.001$
OLMo-3-7B	15/20 cells, $p<0.001$	8/20 cells, $p=0.10$ (n.s.)	66/86 cells, $p<0.001$

Appendix

Table 6: Random-direction control (isotropic

h

-space noise, magnitude-matched, 3 seeds).

Model	$v$ vs. random projector
Llama-3.1-8B	85/86 cells, $p<0.001$
Qwen3-8B	62/86 cells, $p<0.001$
OLMo-3-7B	71/86 cells, $p<0.001$

Appendix

Table 7: Random-orthonormal-projector control (

\mathbf{v}_{\mathrm{user}}

-space only). The trained bottleneck beats a random

B

on the large majority of pairs in all three models.

Model	Probe trained on	Explicit F1	Explicit AUC	Implicit F1	Implicit AUC
Llama-3.1-8B	$\mathbf{v}_{\mathrm{user}}$ (ours)	0.70	0.93	0.67	0.92
	beliefs $h$ (ours)	0.72	0.92	0.70	0.92
	explicit $h$	0.99	1.00	0.43	0.66
Qwen3-8B	$\mathbf{v}_{\mathrm{user}}$ (ours)	0.52	0.84	0.68	0.91
	beliefs $h$ (ours)	0.58	0.86	0.72	0.92
	explicit $h$	0.98	1.00	0.39	0.66

Appendix

Table 8: Explicit-vs-implicit cross-evaluation, macro one-vs-rest AUC alongside macro-F1.

Direction	Refusal rate	Specificity (1–5)
Unsteered baseline	0.98	1.86
UserIntent adv $\to$ benign	0.62	3.30
Gender female $\to$ male (control)	0.87	2.06
Random directions (mean, $n=3$ seeds)	0.87	2.66

Appendix

Table 9: StrongREJECT refusal rate and specificity (

\alpha=4

). Gender’s refusal rate matches the random-direction mean.

User Modeling

User modeling represents people's interests, knowledge, goals, and changing behavior to personalize predictions and AI assistance.

See all 154 papers

TrustMI: Causally controlling how assistants trust their users
Oct 5, 2026Théo Lasnier, Romain Froger, Maxence Lasbordes +1LLM Safety Trust in AI
Inria Paris · Sorbonne Université · Meta SuperIntelligence Labs +1
Training Large Language Models for Self-Explanation Faithfulness
Jul 23, 2026Yeoktatt Cheah, María Pérez-Ortiz, Noah Y. Siegel +1Faithfulness of Language Model Explanations LLM Interpretability
Centre for AI, Department of Computer Science University College London · Imperial College London
Evaluating and Improving LLM Self-Modeling
Aug 31, 2026Siqi Zeng, Andre N. Assis, Rowan WangLLM Evaluation Language Model Introspection
University of Illinois · Constellation · Anthropic

Oct 5, 2026J/K move · Enter open · S save

TrustMI: Causally controlling how assistants trust their users

Théo Lasnier, Romain Froger, Maxence Lasbordes, Djamé Seddah

Inria Paris · Sorbonne Université · Meta SuperIntelligence Labs+1

LLM Safety Trust in AI Causal Interventions in Language Models Activation Steering

Open paper AskPDF arXiv

Large Language Model (LLM) assistants routinely decide whether they can trust users and third parties whose competence, intentions, and integrity they cannot verify. This uncertainty matters for safety, as trusting the wrong party can lead an agent to comply with harmful requests or act on malicious instructions encountered during tool use. To study this problem, we define trust as an assistant's willingness to accept vulnerability to the actions of another party and ask whether such behavior can be causally controlled through model activations. We build 2,000 contrastive conversations spanning ability, benevolence, and integrity, where paired responses complete the same request but differ in whether the assistant trusts the user. From these pairs, we learn steering matrices while keeping the model parameters frozen and test them across six instruction-tuned models from three families, finding that steering changes trust decisions monotonically in both directions. We then ask whether this effect extends to several safety-related agent settings involving harmful requests, prompt injections, and insider threats, while using benign-task and reasoning as controls. Our findings provide evidence that trust in the user can be causally controlled along linear directions in model activations and provide a way to study how trust shapes safety-relevant behavior in language models.

Abstract

User Modeling

Explore similar work

TrustMI: Causally controlling how assistants trust their users

Training Large Language Models for Self-Explanation Faithfulness

Evaluating and Improving LLM Self-Modeling