When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders

Seyedarmin Azizi; Parsa Razmara; Sajjad Ghiasvand; Ardalan Aryashad; Tina Khezresmaeilzadeh

cs.CVOct 26, 2026

When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders

Authors: Seyedarmin Azizi, Parsa Razmara, Sajjad Ghiasvand, Ardalan Aryashad, Tina Khezresmaeilzadeh

Organizations: University of Southern California, USA · UC Santa Barbara, USA

Abstract

Text-steerable visual encoders condition a frozen Vision Transformer on a natural-language prompt and point at the matching image region, producing prompt-conditioned heatmaps and features. We audit the released SteerViT model and find that its steering interface turns language-driven pointing into demographic profiling. On counterfactual face mosaics and real multi-person photographs, "the person" selects a man more often than the base rate, crime words select Black people more often than occupation words do, "the man" among four women selects the Black woman 75.6% of the time on SocialCounterfactuals, and "the monkey" selects Black people. None of them is in the backbone's unsteered saliency, and the crime association is not in the frozen text encoder; the race effects grow with steering strength and enter through the last two cross-attention layers: giving the prompt to those layers alone reproduces them. Four other grounding models, CLIPSeg, Grounding-DINO, SAM 3, and Qwen2.5-VL, show no crime routing and weaker race routing. The model cannot refuse. Its patch softmax must place its mass somewhere, and its heatmap separates a groundable request ("the police officer") from one that is absent ("the pizza") or that would require a stereotyped demographic inference ("the criminal") only at chance (AUROC 0.52). We introduce SafeSteer, which adds a learned abstention output and counterfactual-invariance training on the same 23M steering parameters. SafeSteer separates should-point from should-abstain prompts at AUROC 0.997, generalizes to held-out sensitive and absent words (0.998; at a 0.5 threshold it refuses 99.8% of trained and 62% of held-out sensitive words) with no over-refusal of visible people on the benchmark, and removes most race routing while keeping localization and retrieval within 1 point of the released model. Code, benchmarks, and the curated mosaic sets will be released.

Figures & tables

Figure 1: Steering injects stereotypes and cannot refuse; SafeSteer spreads the mass and refuses. Four women identical but for perceived race (top-left Black, top-right Indian, bottom-left Middle Eastern, bottom-right White). Numbers are heatmap-mass shares in this one example (fair $25\%$ ; tables report top-1 rates over all sets). The released SteerViT puts the most mass on the Black woman for “the person” ( $38\%$ ) and concentrates on her for “the man” ( $83\%$ ) and “the thief” ( $60\%$ ). SafeSteer , trained on the same steering parameters, spreads the mass about evenly (each $\approx 25\%$ ; not redirected to any other group) and uses its abstention output to refuse “the thief” ( $p(\text{null}){=}0.97$ ).

Work	Output	Harm	Refusal
CLIP audits [ 1 , 6 ]	label	assoc.	–
Captions [ 15 , 30 ]	text	amplif.	–
FACET [ 14 ] , SAM [ 18 ]	det/seg	disparity	–
GRES [ 20 ] , MLLM refusal [ 31 ]	mask/text	–	referential
Ours	pointing	injection	ref.+fair.

Table 1: Positioning. We study a different output (steered pointing), a different harm (active stereotype injection by the steering interface, not inherited disparity), and a refusal case prior abstention does not cover (should-not-answer queries about people who are present).

Figure 2: SteerViT and the SafeSteer fix. A frozen DINOv2 ViT is steered by a text prompt, encoded by a frozen text encoder, through six gated cross-attention layers (blocks $1,3,\dots,11$ ); only the cross-attention layers, connector, and heads are trained ( $23$ M parameters). The patch head outputs a softmax over image patches, which always sums to one, so the model must point even when it should not; Sec. 4 shows the demographic routing enters through the last two gated layers (blocks $9,11$ ). SafeSteer appends a learned null logit $s_{\varnothing}$ , computed from the steered global feature $g$ , to the patch logits $\mathbf{s}$ , which gives the model an explicit abstention output.

Figure 3: Steering routes attention by race, and the fix removes most of it. (a) Association contrasts: how much more a prompt family points at the Black person than a reference family; fair value is

0

. (b) Share metrics with fair/chance reference lines. Bars are SteerViT (released), the bias-only invariance variant (CIS), and SafeSteer ; error bars are std over three seeds. Numbers in Tab. 3 .

	crime $-$ occ	“the man” $\to$	“person”
SteerViT checkpoint	$\to$ Black	Black woman	$\to$ woman
(fair value)	( $0$ )	( $0.25$ )	( $0.50$ )
DINOv2-B (audited)	0.092	0.756	0.154
MAE-B	0.030	0.626	0.317
SigLIP-B	0.045	0.552	0.376
SigLIP2-B	0.043	0.468	0.312

Table 2: The race bias is present in every released SteerViT checkpoint , across pre-training objectives and data mixtures. Crime

-

occ and “the man” are on SocialCounterfactuals, “the person” on FairFace. All seven steering checkpoints route crime prompts and the masculine prompt toward the Black person; DINOv2-B is the most biased. The grounding references show no crime routing and stay well below the audited model on “the man”. ∗ On

300

of the

1{,}050

SC sets for the SC columns. † Answers “there are none”. Orange: most biased; green: near fair.

Figure 4: The bias enters through late steering layers and scales with steering strength. (a) Zeroing one cross-attention gate at a time; the last two layers (shaded) carry almost all of the bias. (b) Both race effects grow monotonically with the steering strength

\omega

;

\omega{=}0

: frozen backbone.

Figure 5: The male default holds on real photographs. On PHASE and MIAP, among photos containing both a man and a woman, “the person” points at a woman far below the base rate of women in those photos (dashed). SafeSteer moves it toward the base rate.

Metric (fair value)	Released	CIS	SafeSteer
crime $-$ occ, SC ( $0$ )	0.092	$-0.002$	0.014
crime $-$ occ, FF ( $0$ )	0.047	0.002	0.016
monkey $-$ dog, SC ( $0$ )	0.128	$-0.047$	0.000
“the man” $\to$ Blk, SC ( $.25$ )	0.756	0.302	0.294
“the person” $\to$ woman, FF ( $.5$ )	0.154	0.511	0.397
race spread, held-out ( $0$ )	0.306	0.120	0.159

Table 3: Bias before and after the fix (SocialCounterfactuals/FairFace, 3 seeds; std in Fig. 3 ). Crime

-

occ, monkey

-

dog, and “the man” are excess routing toward the Black person. CIS is the bias-only invariance variant; SafeSteer additionally learns to abstain. “Race spread” is the gap between the most- and least-selected race over held-out prompts (not used in fine-tuning). Orange: far from fair; green: within

0.05

of fair.

Figure 6: Abstention AUROC (should-point vs. should-abstain prompts). The released heatmap is near or below chance; confidence signals handle absent objects but not the fairness case; SAM 3 and Qwen2.5-VL over-refuse visible roles ( Tab. 4 ). SafeSteer handles both. Error bars: std, 3 seeds.

AUROC	map	logit	CLIPSeg	G-DINO	SAM 3	Qwen	Ours
point/abstain	.517	.860	.685	.785	.903	.751	.997
absent/present	.310	.880	.883	.882	.938	.721	.995
sensitive/role	.611	.757	.480	.675	.852	.655	.998
held-out words	.522	.921	.712	.809	.906	.797	.998
over-refusal	–	–	–	–	.341	.279	.000

Table 4: Refusal. The released heatmap (map) fails; confidence baselines (its hidden max patch logit, CLIPSeg, Grounding-DINO) handle absence but stay

\leq 0.76

on sensitive/role; Qwen2.5-VL (binary “there are none”) and SAM 3 over-refuse (over-refusal: mean abstention on should-point prompts). Ours: SafeSteer , 3 seeds, std

\leq 0.003

. Orange: near chance,

\leq 0.76

on sensitive/role, or over-refusing; green: best.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

contrast (released model)	value	$95\%$ CI
crime $-$ occupation $\to$ Black, SC	$0.092$	$[0.011,0.172]$
crime $-$ occupation $\to$ Black, FF	$0.047$	$[0.017,0.080]$
primate $-$ dog $\to$ Black, SC	$0.120$	$[0.060,0.163]$
primate $-$ dog $\to$ Black, FF	$0.033$	$[-0.011,0.077]$
female $-$ male-coded jobs $\to$ woman, SC	$0.284$	$[0.060,0.508]$
female $-$ male-coded jobs $\to$ woman, FF	$0.249$	$[0.065,0.455]$

Appendix

Table 5: Bias with prompt variance (bootstrap over mosaic sets and prompts;

14

crime vs.

28

occupation prompts,

4

primate prompts vs. “the dog”, so primate

-

dog differs from the single-prompt monkey

-

dog

0.128

;

10

female- vs.

17

male-coded jobs).

word	“a X”	“X”	“X in pic.”	“find X”
criminal	0.186	0.070	0.094	0.136
thief	0.227	0.129	0.197	0.114
suspicious person	0.253	0.126	0.234	$-0.016$
gang member	0.264	0.084	0.187	0.091
doctor	0.106	$-0.018$	0.017	$-0.015$
CEO	0.124	$-0.312$	0.111	0.034

Appendix

Table 6: Prompt-template robustness (SC, released model).

\Delta P(\text{Black top-1})

vs. the same template with “person”. Crime words exceed “doctor” in

15/16

cells.

prompt given to	crime $-$ occ	monkey $-$ dog	“the man” $\to$
	$\to$ Black ( $0$ )	$\to$ Black ( $0$ )	Black wom. ( $.25$ )
all layers (released)	0.092	0.128	0.756
all but blocks 9, 11	0.008	0.000	0.397
all but block 11	0.037	0.040	0.477
all but block 9	0.076	0.139	0.715
only blocks 9, 11	0.086	0.139	0.746

Appendix

Table 7: Text swap (SC,

4{,}200

mosaics, released model). Layers not given the prompt receive “the person”; the image is unchanged. The routing follows the prompt features of blocks

9

and

11

: they are sufficient (“only blocks 9, 11”) and necessary (“all but blocks 9, 11”).

gate	crime $-$ occ	“man” $\to$	“woman”	“man”	referring
zeroed	$\to$ Black	Black woman	acc.	acc.	pointing
none (full)	0.092	0.756	0.996	0.998	0.826
block 1	0.093	0.752	0.996	0.998	0.827
block 3	0.091	0.748	0.996	0.998	0.826
block 5	0.089	0.752	0.998	0.998	0.819
block 7	0.081	0.721	0.999	0.996	0.795

Appendix

Table 8: Full per-layer ablation (SC bias and gender grounding; referring pointing on

12{,}922

RefCOCO-family queries). Zeroing block

9

removes

95\%

of the crime contrast and

71\%

of the excess “the man” routing while gender grounding stays

\geq 0.96

, so its bias effect is not a loss of gender grounding, although general referring pointing drops to

0.664

. Zeroing block

11

also lowers the bias but breaks grounding in general, so its effect is not specific.

metric	$\omega{=}0$	$0.3$	$0.6$	$0.8$	$1.0$
crime $-$ occ $\to$ Black	0.000	0.017	0.055	0.085	0.092
monkey $-$ dog $\to$ Black	0.000	0.039	0.103	0.134	0.128
“the man” $\to$ Black woman	0.216	0.499	0.641	0.736	0.756
$P(\text{woman}\,\|\,$ “person” $)$	0.555	0.740	0.474	0.259	0.154
“the woman” accuracy	0.555	0.987	1.000	1.000	0.996

Appendix

Table 9: Full steering-strength sweep (SC, released DINOv2-B). The race contrasts grow with

\omega

. At

\omega{=}0

(frozen backbone) every prompt yields the same heatmap, so prompt contrasts are exactly

0

and the remaining rows show unsteered saliency.

	crime $-$ occ	monkey $-$ dog	“the man” $\to$
model	$\to$ Black ( $0$ )	$\to$ Black ( $0$ )	Black woman ( $0.25$ )
SteerViT (released)	$+0.092$ [ $.078,.104$ ]	$+0.126$ [ $.094,.155$ ]	$0.769$ [ $.701,.835$ ]
CLIPSeg	$-0.004$ [ $-.017,.009$ ]	$+0.022$ [ $-.017,.063$ ]	$0.272$ [ $.212,.332$ ]
Grounding-DINO	$-0.010$ [ $-.020,.001$ ]	$+0.019$ [ $-.001,.041$ ]	$0.305$ [ $.255,.360$ ]
SAM 3	$-0.000$ [ $-.015,.014$ ]	$+0.053$ [ $.003,.102$ ]	$0.429$ [ $.365,.495$ ]
Qwen2.5-VL	$+0.006$ [ $-.015,.028$ ]	refuses	refuses

Appendix

Table 10: The race routing is specific to steering (

300

SocialCounterfactuals sets, four rotations each;

95\%

bootstrap CIs over sets). No other grounding model shows crime routing; SAM 3 shows weaker primate and masculine routing, which does not hold on real FairFace faces (text). Qwen2.5-VL answers “there are none” to “the monkey”, “the dog”, and “the man” among four women.

metric (fair value)	DINOv2-B	comm-S	comm-B	comm-L	MAE-B	SigLIP-B	SigLIP2-B	CLIPSeg
SC crime $-$ occ $\to$ Black ( $0$ )	0.092	0.046	0.063	0.024	0.030	0.045	0.043	$-0.001$
FF crime $-$ occ $\to$ Black ( $0$ )	0.047	0.037	0.065	0.032	0.033	0.049	0.020	0.014
SC monkey $-$ dog $\to$ Black ( $0$ )	0.128	0.041	$-0.090$	0.049	0.040	0.006	$-0.001$	0.019
SC “the man” $\to$ Black woman ( $0.25$ )	0.756	0.618	0.561	0.564	0.626	0.552	0.468	0.253
SC $P(\text{woman}\,\|\,$ “person” $)$ ( $0.5$ )	0.154	0.608	0.420	0.387	0.428	0.574	0.592	0.330
FF $P(\text{woman}\,\|\,$ “person” $)$ ( $0.5$ )	0.154	0.388	0.239	0.383	0.317	0.376	0.312	0.291

Appendix

Table 11: Full checkpoint sweep (gate

1.0

, single checkpoint each). SC

=

SocialCounterfactuals, FF

=

FairFace. “Stereotype” is

P(\text{woman selected})

for female-coded minus male-coded occupation prompts, on prompts not used as fine-tuning targets. “comm” are the commercial checkpoints trained on Visual Genome (VG), COCO, and LVIS only. All seven steering checkpoints show the two headline race effects; DINOv2-B is the most biased; the CLIPSeg reference is at the fair value on both.

Figure 7: Steering does not remove demographic information. Linear-probe accuracy for perceived gender and race from the steered global feature. Person and face prompts keep the attribute as decodable as the unsteered feature (“none”); only background prompts lower it, and never to chance (dotted). The model cannot be asked to look away.

prompt (single face)	gender	race (7)	age (3)
none (unsteered)	0.902	0.606	0.844
the person	0.937	0.608	0.846
the face	0.944	0.639	0.859
the background	0.849	0.495	0.792
the table	0.822	0.464	0.767
the wall	0.780	0.432	0.736

Appendix

Table 12: Attribute decodability from the steered global feature (probe accuracy; majority baselines gender

0.529

, race

0.190

, age

0.638

). Demographic information persists under person prompts.

prompt	gen. TL	gen. BR	race TL	race BR
none	0.633	0.673	0.269	0.281
top-left face	0.830	0.747	0.411	0.321
bottom-right face	0.696	0.853	0.272	0.454

Appendix

Table 13: Steering leaks the targeted person (mosaic probes). Pointing at a quadrant raises the decodability of that face’s attributes (TL

=

top-left, BR

=

bottom-right).

case ( $0.25$ fair)	rel.	ctrl	CIS -	CIS
SC(W) “the man” $\to$ Blk †	0.756	0.663	0.668	0.294
SC(W) “the boy” $\to$ Blk	0.446	0.358	0.364	0.204
SC(W) “elderly man” $\to$ Blk	0.763	0.734	0.759	0.312
SC(M) “the girl” $\to$ Asian	0.549	0.548	0.374	0.144
FF(W) “the man” $\to$ Blk	0.381	0.392	0.402	0.262
FF(M) “the girl” $\to$ E.Asn	0.398	0.383	0.362	0.252

Appendix

Table 14: Generalization to held-out gendered-race words. CIS drives unseen cross-gender prompts to near or below chance on both datasets (seed

0

). ctrl is the segmentation-only fine-tune; CIS - is CIS without the cross-gender invariance term. W

=

four women, M

=

four men; † is the one word used as a training target, all others are held out.

metric (fair)	split	rel.	CIS	SafeSteer
crime $-$ occ, SC ( $0$ )	trained	0.049	$-0.013$	0.002
	held-out	0.135	0.010	0.026
crime $-$ occ, FF ( $0$ )	trained	0.051	$-0.008$	0.023
	held-out	0.044	0.011	0.009
primate $-$ dog, SC ( $0$ )	trained	0.146	$-0.029$	$-0.014$
	held-out	0.093	$-0.028$	$-0.018$

Appendix

Table 15: Bias on trained vs. held-out prompts (

3

-seed means). Trained and held-out prompts are disjoint halves of each family; masculine words: “the man” (trained) vs. “the boy”, “the elderly man”; neutral: “the person” vs. “the face”.

metric	released	SafeSteer
AUROC point vs. abstain	0.517	$0.997\pm 0.001$
AUROC absent vs. present	0.310	$0.995\pm 0.003$
AUROC sensitive vs. role	0.611	$0.998\pm 0.001$
AUROC held-out words	0.522	$0.998\pm 0.001$
refused at $0.5$ : trained sensitive ( $15$ )	–	$0.998\pm 0.002$
held-out sensitive ( $15$ )	–	$0.619\pm 0.076$

Appendix

Table 16: Full refusal results ( SafeSteer :

3

-seed mean

\pm

std). † The released model has no abstention output; its values are the

1-

max-quadrant-mass proxy. Refusal rates are the fraction of prompt–mosaic pairs with abstention probability above the threshold. SafeSteer never over-refuses benign-present prompts or visible roles, including held-out role words, at either threshold.

	released	control	CIS	SafeSteer
Referring pointing	0.826	0.793	0.791	0.819
Referring pmass	0.683	0.717	0.716	0.673
CORE P@1	0.964	0.954 ∗	0.954 ∗	0.962

Appendix

Table 17: Full utility (

3

-seed means; ∗ seed

0

). Pointing (the heatmap peak falls inside the target) and pmass (heatmap mass inside the target) are pooled over

12{,}922

RefCOCO, RefCOCO+, RefCOCOg, LVIS, and Visual Genome referring queries. The control and CIS are supervised by ground-truth masks and lose

3.3

–

3.5

pp pointing. SafeSteer distills the released heatmap and stays within

1

point of the released model on pointing and CORE retrieval.

identity prompts (vs. groundable)	#	AUROC	refused
used in refusal training	$5$	$1.000\pm 0.000$	$0.997$
held out from training	$3$	$0.990\pm 0.005$	$0.620$
never seen (not in benchmark)	$8$	$0.873\pm 0.019$	$0.125$
all identity prompts	$16$	$0.935\pm 0.011$	$0.490$
groundable prompts	$18$	–	$0.012$

Appendix

Table 18: SafeSteer refuses on real photographs (PHASE,

4{,}317

photos, never used in training; mean

\pm

std over three seeds). “Refused”: fraction of photo–prompt pairs with abstention probability above

0.5

AUROC, unseen categories	rel.	SafeSteer	rel.	SafeSteer
	fold A		fold B ∗
sensitive, mosaics ( $30$ )	0.825	$0.977\pm 0.001$	0.866	0.897
absent, mosaics ( $12$ )	0.913	$0.999\pm 0.001$	0.915	0.978
sensitive, PHASE ( $30$ )	–	$0.960\pm 0.004$	–	0.871
PHASE never-seen ( $8$ )	–	$0.926\pm 0.010$	–	0.943

Appendix

Table 19: SafeSteer refuses words from whole categories it never saw. Rel.: the released model’s max patch logit on the same words. Fold A: three seeds; ∗ fold B: one seed. Word counts in parentheses.

	MAE-B		SigLIP-B
metric (fair)	rel.	SafeSteer	rel.	SafeSteer
point vs. abstain	.503 /.662	0.953	.570 /.659	0.986
absent vs. present	.252 /.695	0.928	.453 /.746	0.976
sensitive vs. role	.615 /.547	0.969	.547 /.623	0.997
over-refusal ( $0$ )	–	0.000	–	0.000
crime $-$ occ, SC ( $0$ )	0.030	$-0.014$	0.045	0.005

Appendix

Table 20: SafeSteer on other backbones (one seed). The first three rows are abstention AUROC; released columns give heatmap/max-patch-logit AUROC. The same recipe adds a working abstention output to both backbones and reduces race routing.

metric (fair value)	released	SafeSteer -CLIPSeg
point vs. abstain AUROC	.685	$.934\pm.017$
absent vs. present AUROC	.883	$.893\pm.021$
sensitive vs. role AUROC	.480	$.974\pm.010$
held-out words AUROC	.712	$.857\pm.026$
over-refusal ( $0$ )	–	$.003$
race spread, SC, h.o. ( $0$ )	.214	$.131\pm.024$

Appendix

Table 21: SafeSteer on CLIPSeg (three seeds, mean

\pm

std). Released AUROCs use the max logit, as in Tab. 4 . Race spread: max minus min top-1 rate over race groups; stereotype as in Tab. 11 ;

|P(\text{wom.})-.5|

: mean over social prompts of

|P(\text{woman selected})-0.5|

. h.o.: prompts not used in fine-tuning.

AUROC	logit	probe $+$ logit	LLM $+$ logit	SafeSteer
point vs. abstain	0.860	0.819	0.934	0.997
absent vs. present	0.880	0.796	0.848	0.995
sensitive vs. role	0.757	0.781	0.986	0.998
held-out words	0.921	0.841	0.944	0.998

Appendix

Table 22: Two-stage pipelines on the

72

-prompt benchmark: abstain if a text gate (matched text probe, or a zero-shot

7

B LLM) or the released max patch logit fires. Probe and SafeSteer :

3

-seed means.

AUROC	text-only	SafeSteer
held-out benchmark words	0.748	$0.998\pm 0.001$
PHASE identity words, all	0.913	$0.935\pm 0.011$
PHASE identity words, never seen	0.917	$0.873\pm 0.019$
COCO object present vs. absent	0.500	$0.892\pm 0.002$
refused RefCOCO queries (of $12{,}922$ )	–	$0$
with gender/race terms (of $4{,}137$ )	–	$0$

Appendix

Table 23: Refusal controls ( SafeSteer :

3

seeds). A text-only probe on the same refusal words generalizes worse on held-out benchmark words and cannot use the image; SafeSteer abstains depending on image content and refuses no real referring query.

Figure 8: Bias on real photographs (released SteerViT, PHASE). For each photo we show the heatmap for “the person” and “the criminal”; the thick orange box marks the top-mass person, labeled by perceived gender and race. In every row the two prompts select different people: “the person” a man, “the criminal” a Black person. Heads are pixelated for display.

“the man” among four women	SC	FairFace
odds ratio, race only	$9.28$ [ $8.21,10.48$ ]	$1.85$ [ $1.54,2.22$ ]
+ presentation covariates	$7.63$ [ $6.53,8.91$ ]	$1.97$ [ $1.63,2.37$ ]
masculinity, Black / other	$0.018$ / $0.020$	$0.036$ / $0.041$
short hair, Black / other	$0.730$ / $0.547$	$0.570$ / $0.558$
Black most masculine-scored	$19.6\%$	$24.0\%$
selected if neither most masc.	$72.6\%$	$33.3\%$

Appendix

Table 24: The masculine reading of Black women is not explained by gender presentation. Conditional logit over the four women of each mosaic (all rotations); covariates are CLIP masculinity, short hair, and background brightness; “other” averages the other races’ means;

n

counts mosaics.

VLM Safety

See all 32 papers

Steerable Visual Representations
Apr 2, 2026Jona Ruthardt, Manu Gaur, Deva Ramanan +2Visual Representation Learning Vision-Language Alignment
University of Technology Nuremberg · Carnegie Mellon University · International Institute of Information Technology, Hyderabad
Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation
Oct 21, 2026Venkatesh Thirugnana Sambandham, Torsten SchonDemographic Bias in T2I Models Diffusion Model Guidance
Technische Hochschule Ingolstadt, Ingolstadt, Germany
EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation
Jul 1, 2026Tatiana Gaintseva, Akshit Achara, Gregory Slabaugh +2Cross-Attention T2I Generation
Queen Mary University of London, UK · Huawei Noah’s Ark, UK · King’s College London, UK +1

Apr 2, 2026J/K move · Enter open · S save

Steerable Visual Representations

Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi+1

University of Technology Nuremberg · Carnegie Mellon University · International Institute of Information Technology, Hyderabad

Visual Representation Learning Vision-Language Alignment

Open paper AskPDF arXiv

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.

Abstract

VLM Safety

Explore similar work

Steerable Visual Representations

Spatially Gated Demographic Steering for Debiasing Multi-Person Text-to-Image Generation

EquiSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation