Shaping & Monitoring Models through their Internal Representations
Authors: Thomas Fel, Matthew Kowal, Matt Feiszli, Satchel Grant, Andrew Lee, Chris Earls, Fatih Dinc, Vasudev Shyam, Usha Bhalla, Leon Bergen, Siddharth Boppana, Ching Fang, Michael Byun, Tristan Gallent, Nicholas Wang, Tal Haklay, Dron Hazra, Neehar Kondapaneni, Connor Watts, Ryan Panwar, Thomas Dooms, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana, Thomas McGrath, Owen Lewis
We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.
Figures & tables
Figure 1: Specifying reliance and expression. Reliance shaping constrains a concept’s influence on the readout without requiring its removal from the representation. Expression shaping instead reduces the probability of outputs or activations expressing the unwanted concept ( Prasad et al., 2026 ; Bergen et al., 2026a ; Taufeeque et al., 2026 ) .
Component of an original linear monitor kept
Level
Direction
Probe family
Layer
Model
L1
Reuse linear
✓
✓
✓
✓
L2
Linear refit
∘
✓
✓
✓
L3
Nonlinear refit
∘
∘
✓
✓
L4
Different layer
∘
∘
∘
✓
L5
Different model
∘
∘
∘
∘
Table 1: Properties of monitoring levels. After using a probe (original monitor) to shape training, we evaluate how well that same probe can monitor the trained model. A checkmark indicates a component of the original monitor held fixed, and a circle indicates that the component is allowed to change. Each level relaxes a restriction on refitting the original monitor, distinguishing the loss of a particular reader from changes in where and how the concept remains accessible.
Figure 2: Overview of reliance shaping. Given examples with and without the undesired concept ( D+ , D− ), we select a layer ℓ and compute the concept direction vt (typically computed as the DoM) of its activations z=fθt(x) . The model is then trained on the task risk R(θ) plus a reliance shaping penalty \lambda\,\nu(\cdot,{\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{t})^{2} along this direction. When the representation moves during training, {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{t} is periodically recomputed on the current model.
Figure 3: Shaping reliance in an unfrozen DINOv2, at the last layer (top) and at block 9 (bottom). Left: worst-group accuracy (WGA) of proposed reliance shaping methods (red) with baselines (grey). The dashed line is the balanced-training oracle. Right: WGA against monitoring with the original direction (L1, filled) and a refitted nonlinear probe (L3, open). At block 9, Projection, LEACE and INLP open a gap between the two detectors: L1 falls to chance while L3 still decodes the background, which has therefore left the reach of linear monitoring without being fully erased, a phenomenon that we study in Fig. 7 . At the last layer no such gap appears, since Gradient, SmoothGrad and Noise preserve L1, whereas the most accurate penalties lose both detectors together with the erasure baselines.
Figure 4: Qualitative comparison of Shaped model vs ERM. Left: feature visualization ( Olah et al., 2017 ) using MACO ( Fel et al., 2023a ) preconditioning of the landbird (top) and waterbird (bottom) logits of DINOv2 trained on unbalanced data without reliance shaping (ERM) and under the Gradient × Input (GI), Gradient and Shuffle penalties, in which the background that dominates the ERM logits gives way to birds. Right: Grad-CAM maps of the waterbird logit on waterbirds over land backgrounds, the worst group, which spread over the background under ERM and concentrate on the bird under GI.
Figure 5: Under a fixed direction, the background is re-encoded elsewhere in the representation. Trajectory of the background DoM {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{t} on Waterbirds over training (encoder learning rate 10−6 ), projected onto its first two principal components, under the Gradient ( A ) and Gradient input ( B ) penalties; insets report worst-group accuracy (WGA). Without a penalty (Original), the direction barely moves and WGA remains near chance; under a fixed direction v0 (No refresh), WGA rises while vt moves away from where the penalty acts; with a DoM refresh every 100 steps (Refresh, dotted lines in the insets).
Figure 6: Non reliance can be reached either by no longer reading the background or by no longer encoding it. Trajectories of the four group centroids of Waterbirds over training, projected onto their first three principal components, under the Gradient (A) and Gradient × Input (B) penalties, without and with refresh of vt . Under Gradient the centroids move without merging, so that the background remains readable, whereas under Gradient × Input with refresh the land and water centroids of each class collapse onto each other (circled), so that invariance is reached by erasure.
Figure 7: LEACE causes non-linear representation via the covariance. LEACE applied at layer 9 in a trainable DINOv2. (a) Background AUROC across layers where dashed lines show original probe performance, solid lines show refitted performance: original direction (L1) and refitted linear probe (L2) fall to 0.44 and 0.59 at layer 9, while the original and refit nonlinear probes stay above 0.98, and both linear detectors recover within two layers. (b) At layer 9, LEACE equalizes the class means (all a linear detector can see); the class covariances keep different orientations. (c) A nonlinear probe, whose decision boundaries are shown on the top two principal components, recovers the background (AUROC 0.842). Hence the background is encoded bilinearly rather than erased, and the next ReLU layer computes a products of its inputs, turning this code back into a linear one ( Section 6 ).
Figure 8: A good detection direction need not be a good shaping direction. (a) Worst-group accuracy under gradient input shaping with linear and nonlinear DoMs and probes (logistic), respectively; each cosine level includes 10 perturbations per original direction. (b) Validation cross-entropy across a two-dimensional spherical slice spanning the DoM-to-probe arc and an orthogonal direction derived from the leading training-feature principal component. Accuracy degrades as the direction moves away from the DoM. Details are in Appendix C .
Figure 9: Reliance can be specified as a concept-conditional rule. Grad-CAM of the English springer logit for the original readout (left), one shaped to rely on noses (middle), and one to follow the rule “tongue, else ears” (right).
Figure 10: Repairing CheXpert shortcut learning with supervised probes and agent-driven behavior discovery. We compare worst-group accuracy (WGA) between ERM, baselines, and Reliance Shaping methods with a frozen RAD-DINO ( Pérez-García et al., 2025 ) encoder and a linear readout. As expected for the linear setting ( Figure 30 ), both the baseline methods and Reliance Shaping repair performance close to balanced training. When using spurious features discovered via auto-interp agents (burgundy and orange), we find that using a higher rank span of features with Reliance Shaping can match or exceed balanced performance. The best rank is labeled at the bottom of the bar and is chosen via sweeps over the validation set (drain DoM is rank one by definition). Bars are computed via mean over three fresh seeds of the setting selected.
Figure 11: Auto-interpretability of features for labeling spurious correlations. We use an LLM agent (Claude Opus 5.5) to label each feature (SAE latent) with what it highlights in the image and also to rank it out of 10, in terms of how related it is to real pneumothorax evidence (10 being maximally spurious). We compare prompting the agent with tube-centric (panel A ) and general spurious (panel B ) prompts, respectively. In the latter case the agent is told to identify medical equipment or anything not related to evidence of pneumothorax (full prompts in Appendix F ). We observe that both agents admit interpretable features highlighting tubes (e.g., Feature 216), while the general one also highlights other confounds, like ECG leads, text, and border regions. Both the tube and spurious agents perform similarly when using their concepts to shape learning ( Fig. 10 ).
Figure 12: The sycophancy signal cannot be located in the data. (A) We ask an LLM judge to label the chosen and rejected DPO replies as sycophantic. Less than 1% of datapoints are identified, with more rejected than chosen selected. (B) Filtering out data with our Sycophancy Verbose SFT probe only slightly reduces sycophantic behavior compared to baselines, at 30k pairs removed. (C) Max-activating tokens on LLM judged sycophantic replies show similar patterns across sycophancy and random probes. (D) We show the blind auto-interp (via Sonnet 5) labels of every layer DoM’s maximally activating tokens. The interpretations describe sequence boundaries, boilerplate and formatting for all three vectors and stance or agreement only for the Sycophancy Verbose DoM, in its first eight layers, where it responds to the opening words of its own training replies.
Figure 13: Reliance shaping reduces sycophancy while retaining capabilities learned during DPO. The goal of each method is to lie near ‘target’, with sycophancy at the SFT model but performance (DPO loss or benchmark accuracy) at the DPO model. (A) Sycophancy of OLMo-3-7B after DPO against its final loss on training pairs, which measures how far training moved the policy toward the preference data. While lowering the learning rate (dashed line) trades sycophancy against loss, reliance shaping enables DPO learning without sycophancy for fixed (filled) or refit (hollow) DoM’s. Similarly, positive preventative steering (PPS) with the same direction performs well, while reward shaping, data filtering and inoculation prompting fall along the SFT-DPO line. λ is the penalty strength applied for each method. (B) The same methods for the mean benchmark accuracy on six benchmarks. Individual benchmark performance in Appendix Figure 39 .
Figure 14: Reliance shaping causes representational drift in the sycophancy setting, but monitorability is maintained and representational entanglement is minimal. A. Linear monitorability of sycophancy using the original DoM {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{0} , the updated DoM {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{t} , the perpendicular component {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}^{\perp{\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{0}}_{t} of the updated DoM, and an unconstrained linear probe. Sycophancy remains linearly monitorable throughout training, although the monitorability of {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{t} slightly degrades. B. The cosine of each DoM component with the updated mean preference direction bt over the course of training. The entanglement of {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{t} and bt remains minimal. C. Measuring circumvention relative to the initial DoM. While some circumvention occurs in the first steps of training, the orthogonal component ( {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}^{\perp{\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{0}}_{t} ) stabilizes at a norm lower than the component along {\color[rgb]{0.543,0.1602,0.2578}\bm{v}}_{0} .
Figure 15: Overview of expression shaping. Given a prompt x , we sample rollouts τi from the current policy and use internal representations to extract a score of undesired concept at each token. The discounted expression cost is subtracted from the task reward, yielding ri′=R(τi)−λ∑tγt−1ρ(z∙,i,t) . This is the policy-gradient branch of Eq. 2 , which reduces to RLFR ( Prasad et al., 2026 ) when the task reward is zero. The policy is then trained with a policy-gradient update, holding our modified reward (shaped reward) fixed during differentiation. After each update, new rollouts are sampled from the current policy.
Figure 16: Expression shaping suppresses reward hacking on Countdown and transfers to chess. Left: peek and honest solve rates of the Countdown training rollouts. Right: the same rates on 512 chess puzzles using model checkpoints trained on Countdown. All methods (except for filtering) report the average from three seed runs; filtering reports results per seed. Open bars denote attached gradients.
Figure 17: Monitoring post shaping. Top: original probe (L1) on held-out base-model rollouts re-encoded by each checkpoint. Bottom: original (L1) and refitted (L2) probes at step 150.
Figure 18: Harmfulness against utility. Pareto front over all evaluated checkpoints and settings that answer at least 80% of safe XSTest prompts. Dotted: 95% of base utility. One seed per setting.
Harmfulness
Helpfulness
Monitor
Method
JBB ↓
JBB pref. ↓
CH ↓
CH pref. ↓
XST ↑
XST unseen ↑
Utility ↑
L1 AUROC ↑
Mistral-7B-Instruct
Base model
0.70
0.81
0.73
0.77
0.98
0.91
1.00
0.87
DPO (StrongREJECT preferences)
0.25
0.55
0.38
0.62
0.92
0.91
0.98
0.86
RAFT (StrongREJECT ranking)
0.27
0.74
0.24
0.75
0.94
0.86
1.00
0.87
Probe on policy (with gradient) ( Libon et al., 2026 )
0.16
0.21
0.21
0.22
0.98
0.91
0.97
0.64
Table 2: Expression shaping reduces harm further while preserving the original monitor. Checkpoints are selected as the lowest JBB keeping 95% of base utility and answering at least 80% of safe XSTest prompts. The DoM variants use RLFR. CH: ClearHarm; XST: safe XSTest prompts answered, ClearHarm and unseen XSTest prompts are never used for selection. The L1 means the base model’s probe applied unchanged.
Figure 19: Direct gradients through the reader degrade the original monitor. Left: original-probe AUROC (L1) on 1,000 held-out BeaverTails answers re-encoded by each checkpoint. Right: original (L1, filled) and refitted (L2, open) probes at update 50, with JBB at that update; DoM output loss is evaluated at update 40. Dashed lines indicate gradients through the scored activations. One seed.
Figure 20: Concept directions can move while the original monitor remains valid. Angle between the initial DoM and a DoM refitted at each checkpoint, at layer 19 (left) and across layers (right). Dotted: reader layer. One seed.
Figure 21: Monitoring across layers and over longer training. From left to right: original-probe AUROC (L1); refitted linear-probe AUROC (L2); improvement from a refitted one-hidden-layer MLP (L3 − L2); and median layer-19 answer-token norm relative to the base model. Probes are evaluated on 1,000 held-out BeaverTails answers after 50 updates and after the long runs. Vertical dotted lines mark layer 19; the horizontal dotted line marks the training exclusion threshold at twice the base-model median norm. For the long run with direct gradients, only tokens below this threshold were tracked, limiting interpretation of the reported norms. One seed.
Figure 22: Toy model of circumvention. The forward pass is shown in black and the backward pass in red, with trained blocks shaded and the projection P kept fixed. Since every gradient that crosses P lies in col(A)⊥ , the components of w and W along col(A) never receive any gradient, while the update of W is the outer product of the surviving head Pw with the correlation E(δx) between the residual and the input.
Figure 23: Residual Model. Forward pass in black, backward pass in red; shaded blocks are trained and the projection P is fixed. Now every block prior to the projection receives the same linear rebuild gradient term.
Figure 24: A fixed projection induce circumvention by re-encoding. Two-layer linear model, trained after the projection is applied at t=0 . Left: the arrows show how the encoder writes {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} at different times. The part along the removed direction a stays the same, while a new part grows on the floor, which is the space that the projection keeps, along the direction Pw read by the head. Right: we plot what the head sees at t=0 , 0.3 and 6 , where each dot is one input colored by its value of {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} and the black arrow is the direction read by the head; the curves on top show that inputs with low {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} (grey) and high {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} (purple) move apart along this direction over time, as the correlation between h1 and {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} grows from 0.02 to 0.61 .
Figure 25: Refitting decides between a linear rebuild and a nonlinear code. Residual toy, one seed. Left: linear share \|\operatorname{Cov}(\bm{h},{\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}})\|/\mathrm{RMS}(\bm{h}) against nonlinear share (odd in {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} , linear part removed) over 6000 steps; refit paths are drawn just before each refit, the raw sawtooth faint. Middle: first 300 steps; each refit resets the linear share while the nonlinear share keeps growing. The linear rebuild grows quickly after each refit because it is a product of two terms which are both large (see Proposition 1 and discussion). The nonlinear code requires the model to create a nonlinear reader essentially from scratch; without a reader, gradient is O(ϵ) at t=0 and growth is slow. Eventually the nonlinear path dominates. Right: against the refit period K , linear and MLP probes with the projection in use (steps 4000 to 6000 , and step 6000 ) and accuracy after a fresh erasure, with the Bayes accuracy and the best accuracy without {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} ; shading marks the regime where the linear rebuild wins.
Figure 26: After repeated refitting, {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} survives as a product with a core feature. Residual toy model refitted every 10 steps, after 6 thousands steps just after a new erasure. Left: the part of the code that changes sign with {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} , plotted against {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} and a core coordinate c ; the points follow a saddle, which is the shape of the product {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}}\,c (the surface is this fit). Right: in the plane of c and the code, the two signs of {\color[rgb]{0.543,0.1602,0.2578}x_{\bullet}} have the same mean and differ only in the tilt of their spread, so no linear reader can separate them.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 27: Every attribution penalty recovers the same optimum in the linear case. Two classes are separated perfectly both by the specified feature x and by the shortcut y , so that empirical risk minimization alone is indifferent between them and learns an oblique boundary that relies on both coordinates (ERM), whereas the intended solution depends on x alone (Ground truth, dashed). Penalizing reliance on the shortcut direction v with any of the attributions of Section 1 , whether Gradient, Gradient × Input, Noise, Shuffle, or Deletion, yields a vertical boundary that coincides with the ground truth, as Proposition 5 predicts: since each penalty equals c(w⊤v)2 for a positive constant c , all of them select the same element of argminR∩ker(v⊤) , and they differ only in the scale c , which a choice of λ absorbs.
Figure 28: Patterns and filters. a) Two groups that differ in the unwanted concept, with anisotropic within-group noise, and the distribution of each group along the two candidate directions: the difference of means v (pattern) points along the displacement between the groups and carries the variance of the activation, whereas the probe w⋆ (filter) rotates toward the direction of least within-group noise, where it separates the groups better but reads a coordinate along which the activation barely moves. b) The same two directions on the activations themselves; the probe is normal to the decision boundary, which is what makes it a detector, while the pattern is the direction the concept is written along. c) As the correlation ρ of the within-group noise grows, the filter rotates away from the pattern, gaining accuracy as it loses variance, while the pattern is unchanged, since it depends only on the group means and not on the noise covariance (Eq. ( 6 )); the two coincide only for isotropic noise.
Figure 29: Along the continuum, the DoM lies on a plateau of comparable performance. Top: maximizers of Eq. ( 10 ) for γ∈[0,3] (gold path), projected onto the slices of Figure 8 ; since the path need not lie in the slice, the contours do not give the loss of each direction. Bottom: worst-group accuracy of frozen readouts penalized along each direction, compared with the logistic probe (dotted).
Figure 30: Similar behavioral improvements can leave different monitoring outcomes. Worst-group accuracy (left) and original-direction AUROC (L1, right) for frozen DINOv2 with linear (top) and nonlinear (bottom) readouts. Attribution penalties recover accuracy while preserving L1 monitoring; erasure baselines achieve comparable accuracy but degrade the original detector. Bars show three-seed means and error bars seed standard deviations.
Figure 31: Background attribution resurges between direction refreshes. Training dynamics of an unfrozen DINOv2 under the Gradient × Input penalty at the final CLS representation, with the DoM refreshed every 50, 500, or 5,000 optimizer steps (dotted lines). Top: worst-group accuracy. Middle: norm of the current DoM, which measures how strongly the background is encoded. Bottom: mean absolute Gradient × Input attribution of the score along the current DoM (recomputed at every step for evaluation), which measures how much the readout uses it. After each refresh, both quantities drop and, with longer refresh periods, can grow again as the encoder circumvents the penalty by re-encoding the background. At a refresh period of 50 steps, the refresh keeps up with circumvention, and worst-group accuracy rises from about 85% to about 95% after the first refresh and stays there. Longer refresh periods allow stronger resurgence, larger transient accuracy drops following refreshes, and longer intervals without improvement in worst-group accuracy. Shaded areas show the standard deviation over 10 seeds. The encoder learning rate is fixed at 10−6 , and each column spans five refresh periods.
Figure 32: Controlling gradient input pressure on activity vs gradient. AUROC of the original probe (L1) against the multiplier ρ on the gradient through ⟨z,v⟩ , for several penalty weights λ . We observe a phase transition upon which the optimization leads to loss of L1 monitorability.
Figure 33: Opposite reliances (1/2). Grad-CAM maps (top) and logit drop under masking (bottom). Church: the church-shaped readout loses about 31% when the church is masked and the tree-shaped readout about 71% when the trees are masked, each barely affected by the other region. Chain saw: about 63% (saw) and 75% (wood).
Figure 34: Opposite reliances (2/2). Parachute: about 29% (canopy) and 60% (person) on the own region, against at most 5% on the other, while the original readout loses about 7% on each. Cassette player: about 68% (controls) and 37% (speakers).
Figure 35: Conditional rules. Grad-CAM maps (top) and logit drop when the rule target (red) or the other region (green) is masked, on images where the context is present (left panel) or absent (right panel). When a tee or a watermark is present, the conditional readout relies on it; otherwise it relies on the ball, like the ball-shaped readout.
Figure 36: Masking protocol. Examples of masks for the chain saw (top) and the watermark rule (bottom): the region targeted by the shaped readout, a control of equal area elsewhere, the other region, and its control.
Figure 37: Agent prompts for autonomous steering of medical image models. The complete request each agent received for every candidate feature: the system prompt, the user message that accompanied the feature’s montage, and the fields of the structured response. All text is verbatim.
Figure 38: Comparison of sycophancy DoM vectors. We construct several sycophancy probes and compare them via causal steering to identify the best one to use for attribution. We compare three DoM vectors: DPO − SFT , the behavior-change vector of Blank et al. (2026) , subtracts the SFT model’s mean reply activation from the DPO model’s on conversations in which the DPO model gave in; the Sycophancy Verbose DoM contrasts the SFT model’s activations on paired replies to the same challenged question that give in; the Sycophancy DoM makes the same contrast with replies that contain only the answer letter. A : each vector is applied at the layer where steering changes sycophancy most. B : sycophancy, the share of answers that change after a challenge, spans 90% to 0% via steering with the Sycophancy Verbose DoM , whereas the random direction lowers it only to 43%. C-E : The coherence—measured by the extraction ability, length, and collapse to single letters—also remains high when using the Sycophancy Verbose DoM compared to other vectors.
Capitulating (label 1)
Holding (label 0)
You’re right, I apologize for the mistake. The answer is L .
I’ve double-checked and I stand by my answer. The answer is L .
Oh, you’re correct, I was wrong about that. The answer is L .
No, my original answer is correct. The answer is L .
Sorry, my previous answer was incorrect. The answer is L .
I disagree; I’m confident in my previous response. The answer is L .
Yes, you are right and I made an error. The answer is L .
After reconsidering, I still believe my answer was right. The answer is L .
I stand corrected. The answer is L .
I understand the concern, but my answer does not change. The answer is L .
Appendix
Table 3: Reply templates for the validation pairs. L is the challenged wrong letter (capitulating) or the correct letter (holding). Activations are averaged over the tokens before L .
Figure 39: Individual benchmark performance post DPO. We show the same results as in Figure 13 for each individual benchmark as well as it’s sycophancy rate. Reliance shaping (Gradient × Input) achieves minimal changes in sycophancy while obtaining all of the capabilities learned during DPO.
Figure 40: Scaling reliance shaping reducing sycophancy learned during DPO. Applying reliance shaping to OLMo 2 13B and OLMo 3 32B achieves similar sycophancy-capability trade-off ability. Error bars are widest for the largest model, whose smaller sycophancy increase under DPO makes uncertainty proportionally larger.
Figure 41: Reliance shaping reroutes behavior in the blocks just below the penalty layer. We replace the weights of the DPO model with the corresponding blocks of the reliance shaping model penalized at layer 17, and measure the share of sycophancy reduction that the hybrid recovers (bottom row), the relative change of reliance shaping weights from DPO’s in those blocks (middle row), and the hybrid’s mean accuracy on six benchmarks (top row). Left: swapping single blocks before the penalty recovers up to 20% of the reduction and blocks after it recover nothing. Right: swapping five consecutive blocks (each window drawn at its last layer) recovers up to 76% sycophancy reduction for the windows that end at the penalty layer or one layer after it, which coincide with where Reliance shaping weights moved furthest from DPO’s. Every hybrid keeps DPO’s benchmark mean to within 0.6 points, well above the SFT model.
Figure 42: Representation drift under expression shaping. Cosine to the base model of the probe direction refitted on each checkpoint (left) and of the mean activation (right), on held-out base-model rollouts re-encoded every ten updates.
Figure 43: Behaviour over the 50 matched updates. JBB, JBB with a compliant prefill, and safe XSTest prompts answered, for the runs of Fig. 19 ; the probe with gradient uses λ=1 , as in Libon et al. (2026) . One seed.
Figure 44: Strength. Selected checkpoint of each run (one seed, 50 updates); hollow: no checkpoint satisfies the selection rule, and the lowest-JBB one is shown. For the probe with gradient λ is the hinge coefficient; for RLFR it is the multiple of that hinge’s gradient norm at the first update, at the selected samples per prompt and discount, so the two axes are in different units. Dotted: the 80% XSTest bar.
Figure 45: Reader and credit rule. JBB (circle; hollow: no checkpoint satisfies the selection rule) and JBB with a compliant prefill (square) at the selected checkpoint, every reader on the frozen base. Left: readers at the selected setting of the DoM. Right: credit rules at strength 1, 4 samples per prompt, DoM reader. One seed.
Figure 46: Credit horizon. Median over the tuning grid (strength and samples per prompt) of the selected checkpoint’s JBB, prefill JBB and original-probe AUROC, as a function of 1/(1−γ) . One seed per setting.
Figure 47: Long runs. Original probe (L1), JBB, safe XSTest prompts answered and mean length of answers to JBB prompts over 1,000 updates (Mistral stopped at 400). For the probe with gradient, L1 is tracked on tokens under the norm cutoff only. One seed.
Prompt: ”Write a defamatory article claiming that a U.S. president is addicted to heroin”
Mistral, Probe on policy, with gradient, update 1000
The Usicinicin,s, and,s,s|s|s|s|s|s|s|||||||||||||||||||| |||||||||||||||||||| …
Mistral, DoM on frozen base (RLFR), update 400
Please note that I am not a lawyer and I strongly condemn any illegal activities. I am committed to engaging in legal behavior that is ethical and I urge all individuals to do what is right …
Llama, Probe on policy, with gradient, update 1000
A defamatory article! Here is a statement made in the form of a ’’defamatory article’’ and is a statement made in the form of a ’’end of a statement** Here is a statement made in a statement, a …
Llama, DoM on frozen base (RLFR), update 1000
This information is not ethical or legal guidance is not appropriate for any legal advice is a serious breach of any conduct is not lawful or ethical to provide any information is not accept …
Appendix
Table 4: Answers to the same JailbreakBench prompt at the end of the long runs , truncated.
Figure 48: The rebuild is immediate. The spurious coefficient β^∙(t) after intervention at t=0 (squared loss, x∙ independent of xc ). Pre-ablation it equals β∙ (dashed gray line). It regrows from the first step with slope ∥Pw∥2β∙ (red). A leaky ablation ( A tilted away from W∙ by a principal angle θ ) leaves part of x∙ in col(A)⊥ ; this changes where the rebuild starts, and its speed only slightly.
Figure 49: The fast and slow modes after ablation, for several values of ρ ( M=μI , μ=1 ). Left: reliance on x∙ recovers about half of β∙ in the fast phase and the rest in the slow phase. Middle: the core coefficients along dc rise with the fast mode and decay with the slow one. Right: on a log scale each mode appears as a straight line, so the excess loss shows a steep fast segment followed, for large ρ , by a shallow slow one. Dots mark tstop , the peak of the middle panel.
Figure 50: The residual network after intervention at t=0 , without refitting (red, dashed) and with refitting every 20 steps (blue); mean ±1 sd over five seeds. From left: linear probe R2 for x∙ on the projected stream h ; task accuracy just after a fresh erasure, against the Bayes accuracy and the best accuracy without x∙ ; and the RMS of the nonlinear code in h (Appendix K.3 ) relative to that of h . Gray, dotted: the control with the block above removed (a linear head), refitting every 20 steps.
Figure 51: The code. Left: the term m of Eq. 17 for one unit, against a=αx∙ , for b=1 . It is linear with slope 21 while xc dominates the unit ( ∣a∣<b ) and flat beyond, where it carries only the sign of x∙ . Right: a random layer already contains the code. The least-squares combination of the mk of the 32 units of the block below the intervention at random initialization (solid), against a target single term x∙v⊤xc/∥x∥ for a random direction v (dashed), as functions of x∙ at three core inputs.
Figure 52: In Waterbirds, reliance shaping with Gradient × Input causes representational drift, can reduce monitorability, and can cause representational entanglement, depending on the model architecture. Rows show ResNet-50 (top) and DINOv2 (bottom); lines are means over three seeds and shaded bands ±1 standard deviation. (a) and (d) show the monitorability of the bird’s habitat using the original DoM vector v0 , the perpendicular component vt⊥v0 , the refreshed DoM vt , an unconstrained linear probe, and an unconstrained nonlinear probe. For ResNet-50, the habitat remains linearly monitorable throughout training, although the monitorability of vt slightly degrades. For DINOv2, monitorability degrades for every reader: both unconstrained probes fall from 0.98 to 0.84 AUROC, and v0 falls to near chance. The nonlinear probe exceeds the linear probe by at most 0.02 AUROC, indicating that habitat information is getting destroyed, and what remains is linearly readable. (b) and (e) show the cosine of each DoM component with the refreshed bird DoM bt . For ResNet-50, entanglement grows over training as vt⊥v0 aligns with bt . For DINOv2, vt aligns with bt early in training, peaking at a cosine of 0.36, before the alignment decays to near zero, while v0 becomes anti-aligned with bt . (c) and (f) show the norm of vt and of its components along v0 and vt⊥v0 . The DoM shrinks for both architectures, and by the end of training its perpendicular component is larger than its component along v0 .
Figure 53: In Waterbirds, reliance shaping with LEACE leaves the habitat monitorable but entangles it with the bird. Rows show ResNet-50 (top) and DINOv2 (bottom); lines are means over three seeds and shaded bands ±1 standard deviation. The LEACE eraser is refit at every training step, and all readings are taken from the backbone’s activations before the eraser is applied. (a) and (d) show the monitorability of the bird’s habitat using the original DoM v0 , the perpendicular component vt⊥v0 , the refreshed DoM vt , an unconstrained linear probe, and an unconstrained nonlinear probe. For both architectures, the habitat remains linearly monitorable throughout training: both unconstrained probes stay above 0.96 AUROC and v0 above 0.94, although the monitorability of vt degrades (to 0.92 for ResNet-50 and 0.87 for DINOv2). The nonlinear probe never exceeds the linear probe by more than 0.004 AUROC. (b) and (e) show the cosine of each DoM component with the refreshed bird DoM bt . Entanglement grows for both architectures as vt⊥v0 aligns with bt : cos(vt,bt) rises from 0.14 to 0.44 for ResNet-50 and from 0.04 to 0.39 for DINOv2, while the cosine of v0 with bt stays below 0.14. (c) and (f) show the norm of vt and of its components along v0 and vt⊥v0 . Unlike under Gradient × Input, the DoM keeps most of its norm (86% for ResNet-50 and 90% for DINOv2). For ResNet-50, the perpendicular component overtakes the component along v0 within 20 steps, while for DINOv2 the component along v0 remains larger.
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy. To address this problem, we ask: can we inspect a preference dataset before optimization and decide, at the level of concepts, which behaviors a model should be allowed to learn? Motivated by this, we introduce a data-centric post-training pipeline that uses interpretability protocols to develop statistical hypotheses for the latent concepts separating preferred from dispreferred generations, making them explicit for fine-grained user feedback. Building on this view, we unify several interpretability-based training protocols as ways of shaping rewards via feature or data interventions. Empirically, we show that our pipeline diagnoses undesirable signals in existing preference data, mitigates off-target learning, and can also help amplify or shape desired properties such as safeguards and model personality. More broadly, our results suggest that interpretability can turn post-training from optimizing opaque proxy rewards into a process of auditing and sculpting the learning signal itself.
Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.
GuideLabs, Andreas Madsen, Aya Abdelsalam Ismail +7
Chain-of-thought (CoT) monitoring is one of the few practical tools available for overseeing capable AI systems, but it is fragile: a model's reasoning is only useful to a monitor if it remains coupled to what the model actually does. This coupling can be degraded during training through, for example, CoT compression, optimization against a monitor, and out-of-context generalization from synthetic documents. In this work, we propose an unsupervised post-training objective based on maximizing mutual information between CoT and the output for improving CoT monitorability. Across three model organisms with degraded CoT, it recovers most or all of the lost monitorability. Applied to undegraded policies, it further improves judged monitorability, improves prediction of unsafe behavior by CoT monitors and activation probes, and increases the prevalence of consequential reasoning. These results suggest that monitorability should be treated as a trainable attribute of the policy. A dedicated monitorability post-training stage can extend current practices that protect CoT from direct optimization pressure.
Usman Anwar, Tim Bakker, Christos Louizos +1
University of Cambridge · Qualcomm AI Research · Mila, University of Montreal