A Bag of Concepts? Testing Role-Filler Binding in the Verbalizable Workspace
Authors: Reya Dawlah, Elizabeth McMillin, Michael Shihong Zhang, Badrish Ananth, Rishi Adi Ruia, Ane Zuñiga, Thomas Jiralerspong
Organizations: University of Texas at Dallas · MiraCosta College · University of Waterloo · University of California, Irvine · University of Washington · Massachusetts Institute of Technology · Mila, Université de Montréal
Runtime alignment monitors built on a verbalizable workspace have been proposed as a way to surface a model's silent cognition before it produces output. Their value rests on the workspace carrying enough structure to distinguish safety-relevant propositions, most of which are relational: "the model plans to deceive the user" and "the user plans to deceive the model" share a concept set and differ only in role assignment. We ask whether the workspace carries the role-filler bindings (which entity fills which role) such a monitor would need, on three frontier language models (Qwen3.6-27B, Gemma-3-12B, Gemma-3-27B-IT), under two lens families (J-lens and R-lens), and with two methods for fitting the role direction (difference-of-means and LRE-gradient). We find a clean dissociation: the workspace carries directions that steer what the model outputs about roles but not the representations the model uses to compute them. Linear probes recover role at near-ceiling from the residual stream but score below a rank-matched random control when reading the workspace, and steering the model along the fitted role direction changes which entity it binds to no more than strength-matched control edits do. Ablating the workspace (zeroing the residual stream's projection onto its ten most active lens directions) damages binding on one of the three models (Gemma-3-27B-IT, holding across three lens variants) and shows no binding-specific damage on the other two, with a rank-matched random subspace as the control; this confirms our design detects binding when present, though scale and instruction tuning are confounded there. A workspace-lens monitor built on these directions would therefore miss a role assignment the model itself represents.
Figures & tables
Figure 1: (Left) Two sentences share a concept set but assign roles oppositely. (Center) A bag-of-concepts workspace preserves the concepts but loses the role assignment; a role-filler binding workspace keeps both. (Right) We show that a linear probe recovers role at near-ceiling accuracy from the residual stream and close to chance from J-space, for all three models.
Decoded from
Qwen3.6-27B
Gemma-3-12B
Gemma-3-27B-IT
Full residual stream
0.989
0.975
0.978
Orthogonal remainder
0.984
0.949
0.974
Random subspace (matched rank)
0.643
0.589
0.590
J-space
0.587
0.573
0.574
Table 1: Within-pair role decoding accuracy at the sentence-final position. Chance is 0.50.
c
Displacement
Pooled mean
Non-participant control
Ratio
64
0.328
−0.0134
−0.0161
0.83
128
0.645
−0.0568
−0.0561
1.01
256
1.255
−0.1809
−0.2161
0.84
Table 2: LRE push coefficient sweep on Gemma-3-27B-IT at the entity token. Ratio is pooled-mean over non-participant control.
Model
Site
Role acc. no edit
Role acc. abl. J
Deficit (J)
Same, random
Edit mag. ratio
Qwen3.6-27B
entity
0.790
0.768
+0.014
−0.009
5.39
Qwen3.6-27B †
final
0.790
0.792
+0.000
−0.007
4.97
Gemma-3-12B
final
0.883
0.883
+0.008
+0.001
0.79
Gemma-3-12B †
entity
0.883
0.736
+0.273
+0.101
0.94
Gemma-3-27B-IT
final
0.990
0.991
−0.001
−0.000
0.88
Gemma-3-27B-IT
entity
0.990
0.740
+0.275
+0.006
0.94
Table 3: Binding-specific deficit at the entity and sentence-final tokens, for Qwen3.6-27B, Gemma-3-12B, and Gemma-3-27B-IT. Rows marked † fail the difficulty-matching precondition (binding and recall baseline accuracies within 0.02 of each other).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Decoded from
Qwen3.6-27B
Gemma-3-12B
Gemma-3-27B-IT
Full residual stream
0.594
0.644
0.601
Orthogonal remainder
0.626
0.603
0.605
Random subspace (matched rank)
0.713
0.613
0.575
J-space
0.524
0.514
0.510
Appendix
Table 4: Leave-one-pair-out cross-pair transfer at the entity token. Chance is 0.50.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Applications built on large language models assign the model a role through its system prompt, intending to shape how it responds for their use case. We introduce a validation framework for extracting and injecting occupational role vectors, activation-level directions that capture how someone in a given occupational role would distinctively respond to a task or question. We validate these vectors through two complementary lenses, on Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. The interpretability lens asks whether each vector is a stable, separable, domain-consistent representation of its role: extracted vectors are highly reproducible across repeated runs and sampling temperatures, remain separable even for very similar roles, and agree more with themselves across in-domain and out-of-domain questions than with other roles' signal. The control lens asks whether injecting a role vector during generation makes the model express the role, and sustain it over a conversation, as well as a role prompt does. On Qwen, injecting the vector alone makes the model express the intended role substantially more often than unsteered, but at best matches a role prompt; on both models, adding the vector to the prompt strengthens the role in single answers but did not slow the role's decline over 30 turns. Together, the two lenses show why both are needed: out of domain the role is expressed only subtly, yet the activations still encode it; and a vector that is stable and separable can still barely steer the model's behavior.
Jose L. Sampedro Mazon, Ilmi Yoon, Nadim Saad
Khoury College of Computer Sciences, Northeastern University
How many assignments can a language model recall before it loses track of which value belongs to which entity? We measure this limit using continuous recall curves for 12 models at or below 3B parameters and a threshold sweep over 30 open models up to 12B. On the continuous curves, the load at which recall falls halfway to chance follows K50=cNα, with α=0.820 and R2=0.73. The broader sweep shows an eightfold range associated with pretraining recipe, although the continuous curves show no detectable recipe effect after controlling for scale, with few modern models in the fit. We derive why interference can lower measured capacity by reducing single-binding recall even when the load-dependent recall profile is unchanged. Direct task training also exceeds the extrapolated zero-shot law, but different measurement criteria prevent interpreting that comparison as a capacity gain. Its formation times follow a power-law form in two independent codebases, conditional on runs that succeed. Together, these results characterize capacity at the model's query interface. Bounds on joint recall and a decomposition of policy errors connect this measurement to working memory and instruction following, without treating recall as a measure of alignment. The controlled task also provides a baseline for testing whether binding limits constrain world-state tracking; the present experiments do not measure state updates or downstream transfer.
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha
A key question for AI safety is whether a language model expresses all of its reasoning in its output tokens. We demonstrate a concrete failure mode where frontier models exhibit invisible reasoning by leveraging semantically irrelevant filler tokens to improve performance on synthetic reasoning tasks. We evaluate 13 frontier language models across three tasks and find that many models benefit significantly from filler tokens, with accuracy improvements of up to 13 percentage points. The benefit depends on which tokens are used and differs across models. We further show that filler tokens enable Claude Opus 4.5 to satisfy a hidden modular arithmetic constraint without sacrificing accuracy on its primary task, demonstrating that invisible reasoning can serve objectives entirely invisible to CoT monitoring. Reinforcement learning gives Qwen3-235B strong preferences over filler token content, but neither RL nor supervised fine-tuning produces a filler token benefit that persists at test time. Our results indicate that frontier models already perform consequential computation with no interpretable trace in their output tokens.
Vatsal Baherwani, Tom Goldstein, Ashwinee Panda
New York University · University of Maryland · TogetherAI