PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering

Trong-Thang Pham; Loc Nguyen; Anh Nguyen; Hien V. Nguyen; Ngan Le

cs.CVMar 7, 2026

PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering

Authors: Trong-Thang Pham, Loc Nguyen, Anh Nguyen, Hien V. Nguyen, Ngan Le

Organizations: University of Arkansas, Fayetteville AR, USA · University of Liverpool, Liverpool, UK · University of Houston, Houston TX, USA

Abstract

Generative diffusion models are increasingly used for medical imaging data augmentation, but text prompting cannot produce causal training data. Re-prompting rerolls the entire generation trajectory, altering anatomy, texture, and background. Inversion-based editing methods introduce reconstruction error that causes structural drift. We propose PolypSteer, a training-free activation-steering framework for endoscopic synthesis. PolypSteer identifies a pathology vector for each contrastive prompt pair in the cross-attention layers of a diffusion transformer. At inference time, it steers image activations along this vector, generating counterfactual pairs from scratch where the only difference is the steered concept. All other structure is preserved by construction. We evaluate PolypSteer across three experiments on Kvasir v3 and HyperKvasir. On counterfactual generation across three clinical concept pairs, PolypSteer achieves flip rates of 0.800, 0.925, and 0.950, outperforming the best inversion-based baseline in both concept flip rate and structural preservation. On dye disentanglement, PolypSteer achieves 75% dye removal against 20% (PnP) and 10% (h-Edit). On downstream polyp detection, augmenting with PolypSteer counterfactual pairs achieves ViT AUC of 0.9755 versus 0.9083 for quantity-matched re-prompting, confirming that counterfactual structure drives the gain. Code is at link https://github.com/UARK-AICV/PolypSteer

Figures & tables

Figure 1: PolypSteer method pipeline. A&N : Add & Norm. FF : Feed Forward. CA : Cross-Attention. SA : Self-Attention. (a) Offline Pathology Vector Estimation: CA features $h_{l,t,z}$ are collected from the frozen DiT for positive and negative prompts across $Z$ random seeds. A Mean step yields $\bar{h}^{pos}_{l,t}$ and $\bar{h}^{neg}_{l,t}$ . a S & N (Subtract & Normalize) step then produces the unit pathology vector $v_{l,t}$ (dyed lifted polyp $\to$ polyp). (b) Inference-Time Steering: Unsteered Inference runs the frozen DiT unmodified. Steered Inference applies Spatially Selective Pathology Steering (SSPS) at layers $l\in\{L_{s},\dots,L_{e}\}$ across all $T$ denoising steps. Inside SSPS, a CSG (Cosine-similarity gate) produces the per-token score $\sigma_{l,t}$ , which is scaled by $\alpha$ and fed to an Update step that subtracts the aligned component from $h_{l,t}$ , yielding the counterfactual activation $h^{\prime}_{l,t}$ . While both inference branches are shown together here, only the desired branch is executed in practice.

		ConvNeXt		ViT
Methods	CF?	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$
Real-only	✗	$0.8977_{\pm.019}$	$0.8469_{\pm.023}$	$0.9062_{\pm.022}$	$0.8942_{\pm.026}$
Re-prompting	✗	$0.9038_{\pm.016}$	$0.8714_{\pm.020}$	$0.9147_{\pm.018}$	$0.9083_{\pm.022}$
PnP	✓	$0.9174_{\pm.014}$	$0.9086_{\pm.017}$	$0.9437_{\pm.013}$	$0.9518_{\pm.017}$
h-Edit	✓	$0.9097_{\pm.015}$	$0.8902_{\pm.019}$	$0.9295_{\pm.016}$	$0.9312_{\pm.020}$
PolypSteer	✓	$\mathbf{0.9263}_{\pm.012}$	$\mathbf{0.9341}_{\pm.015}$	$\mathbf{0.9591}_{\pm.011}$	$\mathbf{0.9755}_{\pm.013}$

Table 1: Downstream polyp detection on HyperKvasir (

N{=}1{,}000

synthetic images per condition). CF: counterfactual generation.

Figure 2: Qualitative comparison across concept pairs. Rows: Unsteered, PnP, h-Edit, and PolypSteer (Ours). Columns show Polyp $\to$ Normal Cecum, Ulcerative Colitis $\to$ Normal Cecum, Esophagitis $\to$ Normal Z-line, and dye steerings.

	(A) Polyp $\to$ Norm. Cecum					(B) UC $\to$ Normal Cecum		(C) Esoph. $\to$ Normal Z-line
Method	Flip $\uparrow$	$\Delta p$ $\uparrow$	Bg-LPIPS $\downarrow$	Bg-SSIM $\uparrow$	Bg-PSNR $\uparrow$	Flip $\uparrow$	$\Delta p$ $\uparrow$	Flip $\uparrow$	$\Delta p$ $\uparrow$
PnP	0.627	0.543	0.1518	0.8506	19.32	0.682	0.617	0.718	0.651
h-Edit	0.714	0.638	0.1565	0.8222	19.51	0.741	0.663	0.796	0.712
PolypSteer	0.800	0.721	0.1449	0.8951	22.95	0.925	0.902	0.950	0.928

Table 2: Counterfactual generation (UC: Ulcerative Colitis, Esoph.: Esophagitis). Bg metrics are not applicable for pairs (B) and (C): the polyp segmentor cannot produce valid masks for diffuse mucosal inflammation or esophageal anatomy. Re-prompting is excluded because it rerolls the trajectory and produces different anatomy.

	UNet++				Segformer
Method	Eff. LPIPS $\downarrow$	Eff. SSIM $\uparrow$	Eff. PSNR $\uparrow$	DDR $\downarrow$	Eff. LPIPS $\downarrow$	Eff. SSIM $\uparrow$	Eff. PSNR $\uparrow$
h-Edit	0.4540	0.0957	2.78	0.900	0.5190	0.0952	2.73
PnP	0.3021	0.1872	4.93	0.800	0.3235	0.1865	4.82
PolypSteer	0.2027	0.6584	15.57	0.250	0.2193	0.6521	15.18

Table 3: Dye disentanglement under two background segmentors. DDR = dye detection rate (lower = dye removed). Eff. LPIPS

=\text{LPIPS}/(1{-}\text{DDR})

(

\downarrow

). Eff. SSIM

=(1{-}\text{DDR}){\times}\text{SSIM}

, Eff. PSNR

=(1{-}\text{DDR}){\times}\text{PSNR}

(

\uparrow

(a) $L_{s}$ – $L_{e}$ ablation
Layers	W $\dagger$	Flip $\uparrow$	$\Delta p$ $\uparrow$
0–8	8	0.025	0.018
8–16	8	0.800	0.721
12–20	8	0.013	0.009
0–12	12	0.377	0.396
8–20	12	0.700	0.623

Table 4: Ablation studies (Polyp

\to

Normal Cecum). (a)

L_{s}

–

L_{e}

ablation (

\alpha{=}2.5

N{=}50

\dagger

W = window width, i.e. number of layers steered. (b) Steering strength (layers 8–16,

N{=}50

). (c) Number of seeds (layers 8–16,

\alpha{=}2.5

Figure 3: Left : Per-token cosine similarity maps $\sigma_{8,t}$ at layer 8 for selected diffusion steps $t\in\{8,12,14,19\}$ (left to right). Warmer colours indicate stronger alignment with the pathology vector. Right : corresponding unsteered (dyed lifted polyps) and steered (normal cecum) endoscopy images.

Counterfactual Image Generation

See all 18 papers

To do( $x$ ) or not to do( $x$ ): Medical Image Counterfactuals for Dataset Augmentation
Sep 12, 2026Yasin Ibrahim, Robin J. Evans, Konstantinos KamnitsasFairness in Medical Image Classification Counterfactual Data Augmentation
Department of Engineering Science, University of Oxford · Department of Statistics, University of Oxford
MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation
Sep 14, 2026Yan Zeng, Changlu Guo, Anders Nymark Christensen +2Counterfactual Explanations Latent Diffusion Models
Department of Applied Mathematics and Computer Science, Technical University of Denmark
Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis
Mar 22, 2026Tian Xia, Matthew Sinclair, Andreas Schuh +8Counterfactual Image Generation Image Generation
Department of Computing, Imperial College London, UK · HeartFlow, Inc., Mountain View, CA, USA

Sep 12, 2026J/K move · Enter open · S save

To do( $x$ ) or not to do( $x$ ): Medical Image Counterfactuals for Dataset Augmentation

Yasin Ibrahim, Robin J. Evans, Konstantinos Kamnitsas

Department of Engineering Science, University of Oxford · Department of Statistics, University of Oxford

Fairness in Medical Image Classification Counterfactual Data Augmentation Synthetic Data Augmentation Counterfactual Image Generation Causal Counterfactual Generation Medical Image Synthesis+1

Open paper AskPDF arXiv

Medical image analysis is often hindered by biased datasets, which can lead to biased models and limited clinical applicability. A promising strategy for mitigating such biases is to augment training data with synthetic images. Counterfactual (CF) generation is one such strategy, though the term is used in two different senses: in some works, CFs are produced through causality-based interventions derived from structural causal models, whereas in others, they are produced by non-causal image edits or conventional conditional generative models, such as altering anatomy or adding pathologies. In this work, we study this distinction and evaluate its practical consequences for medical image augmentation. We compare three conditioning strategies:

\textit{Deterministic}

, which changes selected variables while holding the remaining variables fixed;

\textit{Undirected}

, which updates variables according to learned statistical associations without assigning causal directions; and

\textit{Causal}

, which propagates interventions along a directed causal graph. We analyse how these choices affect the resulting images, and explore when causally grounded methods improve dataset augmentation or bring limited benefit. In particular, we assess downstream performance and fairness, where fairness refers to reduced sensitivity to dataset biases across sensitive subgroups. Our experiments demonstrate that using a causal approach to synthetic training data generation can lead to tangible benefits, with these insights offering valuable guidance to machine learning practitioners for the effective design of data generation protocols.

Abstract

Counterfactual Image Generation

Explore similar work

To do(xxx) or not to do(xxx): Medical Image Counterfactuals for Dataset Augmentation

MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation

Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis

To do( $x$ ) or not to do( $x$ ): Medical Image Counterfactuals for Dataset Augmentation