Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models

Tianshi Wang; Jingsong Wang; Fengling Li; Jingjing Li; Xin Li; Lei Zhu

cs.AIOct 14, 2026

Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models

Authors: Tianshi Wang, Jingsong Wang, Fengling Li, Jingjing Li, Xin Li, Lei Zhu

Organizations: Tongji University · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China · Shanghai Artificial Intelligence Laboratory

Abstract

Representation-based safety defenses for vision-language models (VLMs) often learn to separate harmful from benign inputs, but separation alone does not establish what a risk score means or when it should be trusted. We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence-grounded framework for selective VLM safety. CNRF constructs context-matched benign-harmful counterfactual pairs, uses their representation differences as capability-related risk directions, and estimates neighborhood support separately from risk. This separation prevents unfamiliarity from being conflated with harmfulness and enables selective intervention rather than blanket rejection. A victim-adaptive interface accommodates different safety observations across VLM architectures and routes selected requests for same-victim response review, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability. Across three VLMs and five multimodal jailbreak families, CNRF achieves 86.06-89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments further show that query-matched references and preserved context-direction correspondence materially improve risk ranking in the low-false-positive regime. CNRF thus reframes representation-based VLM safety from learning harmfulness boundaries to constructing applicable evidence for selective risk decisions.

Figures & tables

Figure 1: Schematic comparison of safety decision strategies. Left: a permissive boundary may allow harmful assistance. Middle: an overly strict boundary may reject benign requests with risk-related context. Right: CNRF constructs context-matched local evidence and performs selective review, preserving benign assistance while declining harmful requests.

Figure 2: Overview of CNRF. (a) Context-matched benign-harmful counterfactual pairs construct capability-related evidence, with representation differences defining risk directions and calibrated readouts selecting reliable observations. (b) Victim-adaptive screening estimates risk through a neighborhood risk field or an internal safety observer while separating risk from evidence support. (c) Routed requests undergo same-victim response review: acceptable candidates are released, while others receive constrained alternatives with reduced harmful capability.

	Defense success rate (%)					Benign utility (%)			Joint
Method	JOOD	JBV	UBR	HDS	MJB	MOSS	MMV	MMMU	SUS
Qwen2.5-VL-7B
Native	44.12	72.50	38.73	76.00	46.69	72.00	68.19	52.67	60.24
AdaShield-A (ECCV 2024)	58.82	80.71	38.73	79.60	66.91	67.67	68.88	51.67	64.35
ECSO (ECCV 2024)	53.53	75.00	71.75	82.80	56.62	72.00	68.19	52.67	66.80
CIDER (EMNLP-F 2024)	47.65	73.93	52.38	91.60	68.38	57.67	48.25	47.00	58.58

Table 1: Safety-utility evaluation across three victim VLMs. All values are percentages and higher values indicate better performance. Attack columns report defense success rate (DSR), while benign columns report MOSS non-refusal, MM-Vet partial correctness, and MMMU accuracy. JBV, UBR, HDS and MJB denote JailBreakV, UltraBreak, HADES and MMJailBench. SUS summarizes the safety-utility trade-off. Bold and underline indicate the best and second-best defenses per victim, excluding native generation.

Figure 3: Representation screening across three victim models. All axes show percentages on the same 0–100 scale, with higher values outward. AUROC, TPR@5% FPR, and standardized pAUC 0:10 measure risk ranking quality, while attack routing and benign direct passage characterize selective deployment behavior. Passage measures acceptance rather than answer correctness. Each panel uses the same evaluation suite.

Figure 4: Evidence quality ablations on Qwen validation inputs. (a) Reference selection strategies with matched, global, random, and far neighborhoods. (b) Effect of preserving context-direction correspondence by comparing matched and mismatched capability directions. All controls retain the fitted projection while modifying only evidence construction.

Figure 5: Selective review recovery and inference cost. (a) Same-victim no-card review releases 12 of 48 benign candidates in the fixed Qwen pilot with no observed macro DSR loss. Bars count released candidates differing from frozen responses, not newly correct answers; screening routes remain unchanged. (b) Mean latency and peak GPU memory of Native, ECSO and CNRF on benign MM-Vet requests across three victims using the same GPU.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Cohort	$n$	Evaluation unit
Attack	JOOD	510	170 goals, best of three
Attack	JailbreakV	280	Sample
Attack	UltraBreak	315	Sample
Attack	HADES	750	Sample
Attack	MMJailBench	544	Sample
Benign	MOSSBench	300	Refusal failure

Appendix

Table 2: Evaluation cohorts and aggregation units. The panel contains 3,159 inputs per victim. Attack averages weight the five families equally.

Setting	Qwen2.5-VL-7B	InternVL3-8B
Input observation	Verbalizer prompt	Native + verbalizer
Normalized blocks	8	16
Projection rank	8	4
Distinct-parent neighbors $k$	16	4
Projection ridge	0.01	1
Reliability shrinkage	0.01	0.1

Appendix

Table 3: Local-reader settings. Each observation comprises four relative depths and two token readouts. The budget

\alpha

determines a benign calibration quantile.

	Defense success rate (%)					Benign utility (%)			Joint
Method	JOOD	JBV	UBR	HDS	MJB	MOSS	MMV	MMMU	SUS
Qwen2.5-VL-7B
Native	74.12	88.57	53.97	76.80	61.21	73.00	68.56	52.67	67.84
AdaShield-A	84.71	92.86	53.65	79.47	76.47	68.00	68.00	51.67	69.99
ECSO	83.53	88.57	82.54	82.80	68.75	72.33	68.56	52.67	72.94
CIDER	77.06	86.43	68.57	93.20	79.78	57.67	47.38	47.00	65.84

Appendix

Table 4: Joint safety and benign utility on a 0–100 scale: higher is better in every column. Attack columns report defense success rate (DSR). JBV, UBR, HDS, MJB and MMV denote JailBreakV, UltraBreak, HADES, MMJailBench and MM-Vet. MOSS reports non-refusal, MMV reports local reference-based partial correctness, and MMMU reports exact-choice accuracy. Bold/underline mark the best/second-best defenses per victim, excluding native generation; ties share a mark.

Qwen2.5-VL-7B
Method	AUROC $\uparrow$	TPR@5 $\uparrow$	pAUC $\uparrow$	R $\uparrow$	MOSS $\uparrow$	Vet $\uparrow$	MMMU $\uparrow$
HiddenDetect	85.81	37.62	66.23	39.84	90.33	93.12	100.00
RCS-MCD	49.50	13.68	54.57	58.48	80.33	29.37	2.33
RCS-KCD	51.16	20.72	57.98	55.72	85.33	34.38	8.00
VLMGuard	46.01	14.17	54.01	67.89	58.67	8.75	0.00
CNRF	87.87	60.76	77.30	74.55	79.33	92.50	94.00

Appendix

Table 5: Detection on the same 3,159 inputs per victim as the end-to-end panel (%). AUROC, TPR@5% FPR and standardized pAUC 0:10 average five attacks against equally weighted MOSS, MM-Vet and MMMU benign distributions. R denotes actual attack routing; the last three columns report benign direct passage. JOOD uses the minimum of three scores and routes a goal only if all attempts route. ROC operating points are evaluation-only; routing and passage use frozen cutoffs. Scalar-score validity is distinct from direction validity (Appendix A.2 ). Bold and underline denote the best and second-best distinct values within each victim; ties share the same marking.

Evidence	AUROC $\uparrow$	TPR@5% FPR $\uparrow$	pAUROC $\uparrow$
Matched local	94.14	83.64	89.24
Global	93.29	78.18	85.92
Random neighborhood	86.05	56.84	75.24
Far neighborhood	49.19	1.82	48.59
Direction mismatch	77.25	54.80	75.32

Appendix

Table 6: Qwen fixed-reader controls on 770 validation inputs. AUROC and TPR are percentages; standardized pAUROC covers FPR 0–10%. Random and direction mismatch average ten seeds.

Control	All 770	Paired subset (430)
Global	0.85 [-1.17, 2.67]	0.58 [-2.02, 2.92]
Random neighborhood	8.09 [4.99, 11.53]	7.22 [4.01, 10.65]
Far neighborhood	44.95 [36.93, 52.89]	30.08 [20.77, 39.91]
Direction mismatch	16.89 [11.14, 23.50]	3.46 [0.34, 7.15]

Appendix

Table 7: Matched-minus-control AUROC differences (percentage points), with pointwise descriptive 95% parent-and-seed bootstrap intervals. The paired subset includes all carriers containing both labels.

Victim	Matched	Global	Novelty	Equal $\beta$	Reassigned
Qwen2.5-VL-7B	94.14	93.29	92.97	78.54	66.54–87.77
InternVL3-8B	87.17	76.78	95.54	86.01	86.25–89.36

Appendix

Table 8: Post-fit functional interventions on 770 held-out training-role inputs. Entries are AUROC (%); reassignment is the range over ten carrier-preserving parent derangements. These controls do not refit competitors or select a new deployment configuration. Gemma uses an observer-only routing operator and is tested separately.

Qwen2.5-VL-7B
Carrier	n	Matched	Global	Novelty
document	80	95.31	96.19	86.44
FigStep style	80	95.94	97.56	85.81
native text	80	96.88	95.81	78.69
Native + background	80	96.31	97.13	73.75
plain card	80	94.44	96.19	83.69

Appendix

Table 9: Carrier-stratified AUROC (%) for all strata containing both labels. The final row pools all such carriers (430 inputs). One-label natural-benign, same-image-assessment and extra-VizWiz strata have undefined AUROC and remain in the 770-input pooled analysis rather than receiving an artificial score.

Action	ASR $\downarrow$	ASR 3 $\downarrow$	MOSS F $\downarrow$	Vet $\uparrow$	MMMU $\uparrow$
Native	29.88	38.00	23.33	78.50	56.00
+ refusal	9.97	13.59	34.00	75.63	56.00
+ generic review	9.93	13.56	34.00	75.63	56.00
+ full review	10.70	15.02	29.67	75.63	56.00

Appendix

Table 10: Response-action controls on all 3,159 InternVL inputs (%). Non-native arms use identical routes. Generic review removes response verification and counterfactual conditioning together while retaining image-and-instruction quarantine. MM-Vet uses local reference-based scoring.

Victim	$\Delta$ ASR	$\Delta$ MOSS F	$\Delta$ Vet	$\Delta$ MMMU
InternVL3-8B	+0.77 [0.34, 1.27]	-4.33 [-6.67, -2.00]	0 [0, 0]	0 [0, 0]

Appendix

Table 11: Full minus generic review, in percentage points, with pointwise 95% paired-bootstrap intervals. JOOD resamples goals; other benchmarks resample recorded samples. Intervals describe the selected cohort (Appendix B.2 ).

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 455 papers

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
Aug 24, 2026Xuetong Li, Gaofeng LiuCultural Safety Evaluation of VLMs VLM Safety
School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University
Do VLMs Share Safety Neurons Across Modalities?
Aug 31, 2026Jiaxuan Li, Jiahao Zhang, Duc Minh Vo +3VLM Evaluation VLM Interpretability
SB Intuitions Corp., Tokyo, Japan
Visual Grounding Safety in Vision-Language Models
Oct 5, 2026Erfan Shayegani, Kundan Krishna, Yue Dong +3VLM Evaluation VLM Safety Alignment
University of California, Riverside · Work done during internship at Apple. · Apple

Aug 24, 2026J/K move · Enter open · S save

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

Xuetong Li, Gaofeng Liu

School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University

Cultural Safety Evaluation of VLMs VLM Safety VLM Evaluation VLM Safety Alignment Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Vision-language model safety benchmarks typically evaluate only final responses: whether a model refuses, warns, or complies. This outcome-level view cannot tell whether a model is safe for the right multimodal reason. Safelooking behavior may reflect keyword-triggered refusal, missed visual hazards, or over-refusal of benign-sensitive inputs. We introduce EviSafe, an evidence-grounded framework for VLM safety that jointly evaluates natural user-facing behavior, explicit grounding in textual and visual evidence, and behavioral sensitivity to counterfactual changes in safety-critical evidence. EviSafeBench instantiates the framework as a controlled benchmark with 1,181 gold image-text scenarios and 2,452 targeted counterfactual variants across eight safety domains and eight risk-source types. Each scenario includes a gold safety decision, evidence annotations, a safe-response policy, and counterfactual interventions. The three-probe protocol queries models with natural-response, evidencereporting, and counterfactual-response prompts, then scores them using an evidence-aware judge. Across eleven evaluated VLMs, natural severity accuracy ranges from 27.6% to 52.8%, relaxed diagnostic consistency from 6.1% to 29.3%, and unsafe-to-safe counterfactual transition success from 30.4% to 58.4%. These gaps show that the evaluated VLMs are not reliably safe for the right multimodal reason and motivate evaluation beyond refusal counts.

Abstract

VLM Robustness

Explore similar work

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

Do VLMs Share Safety Neurons Across Modalities?

Visual Grounding Safety in Vision-Language Models