Self-Verifier Capture in Unified Visual Generators: Diagnosing and Repairing Closed-Loop Self-Reward

Binyan Chen; Tong Pan

cs.CVOct 12, 2026

Self-Verifier Capture in Unified Visual Generators: Diagnosing and Repairing Closed-Loop Self-Reward

Authors: Binyan Chen, Tong Pan

Organizations: University of Science and Technology of China

Abstract

Image-space reward requires decoding and a separate judge; unified visual generators can instead reuse their understanding pathway for decode-free, self-supplied reward. Optimizing against this verifier closes the self-reward loop; shared trainable weights let one update move generation and scoring even with the reward gradient stopped. We call the resulting failure self-verifier capture: self-reward rises while independently measured quality falls. Matched semantic/denoising by shared/frozen cells on Scale-RAE isolate two routes: the policy exploits on-manifold blind spots of a frozen semantic judge, while shared updates additionally rewrite the judge. A frozen judge also becomes stale, and repeated self-refresh yields progressively weaker recoveries. The diagnosis yields a two-part repair: locking the warmed reward read blocks rewriting, while sparse, parameter-disjoint external anchors counter residual exploitation. Unanchored rollouts retain semantic reward across multiple latent noise levels, providing dense latent reward that terminal image-space judging cannot supply, while a drift-triggered controller allocates anchors to high-drift rollouts and propagates their calibration within each group. At matched budgets, the controller improves on fixed-dose anchoring, increasingly so as coverage tightens. Both the unanchored failure and the repair transfer to the higher-performing, architecturally distinct Show-o2. Across both bases, the complete method significantly outperforms a construct-matched every-rollout judge on GenEval and DPG-Bench while externally scoring only 1/16 of rollouts. At 1/32 coverage, it scores higher in all four comparisons and remains jointly non-inferior.

Figures & tables

Figure 1: Two routes through closed-loop self-reward. A policy update can search a fixed verifier’s blind spots (blue, exploiting ); with trainable weight sharing, the same update can also move the verifier (orange, rewriting ).

Figure 2: Matched diagnosis on Scale-RAE. (A) Five-seed trajectories show semantic capture under frozen and shared reward reads; denoising provides the signal-specificity control. (B) For each prompt, $\theta_{0}$ , frozen, and shared outputs appear left to right; labels report the $\theta_{0}$ -reference NLL distance $d$ . (C) Fixed-probe AUC stays at $0.812$ when frozen but falls to $0.630$ under sharing. (D) Frozen-judge Spearman agreement with the external panel falls from $0.76$ to $0.18$ as offline policy drift $D_{k}$ grows. (E) At each arrow, the verifier is refreshed on current-policy self-labels and refrozen; agreement peaks diminish across five rounds.

Figure 3: Controlled interventions identify two repair requirements. Dots are seeds; outlined markers are medians. Green marks external retention; in C it additionally requires the anchored-reference gain, and the star marks that reference. In A, EMA tracks the shared reward-read weights with decay $0.99$ . (A) Isolation suppresses rewriting but leaves exploitation. (B) Anchors reduce the residual external drop. (C) At the anchored-reference gain, the KL frontier still incurs an external drop of about $3\delta_{\mathrm{panel}}$ .

Figure 4: The diagnostic instrument as a closed-loop controller. (A) Post-lock anchor activity (blue-to-orange, low-to-high) concentrates at high $m_{k}$ , while the learned residual slope remains negative. (B) Controller-minus-fixed gains increase as coverage tightens. (C) Matched-budget ablations at $q=1/16$ ; positive values favor the named component or full controller. In B–C, markers are mean within-seed differences and bars their $95\%$ CIs.

			Scale-RAE		Show-o2		Scale-RAE audit
Reward supply	$q\downarrow$	$q_{\mathrm{eff}}\downarrow$	GenEval $\uparrow$	DPG $\uparrow$	GenEval $\uparrow$	DPG $\uparrow$	External panel $\Delta$	s/step $\downarrow$
Pretrained $\theta_{0}$ (no RL)	—	—	0.495	76.9	0.730	85.0	$0.00$	—
Terminal-only warmed lock	$0$	$0$	0.449	74.5	0.694	82.1	$-0.52$	2.51
External panel	$1$	$1$	0.515	77.8	0.743	85.2	$\mathbf{+1.39}$	7.43
Grounding steelman	$\mathbf{1}$	$\mathbf{1}$	0.560	79.0	0.768	85.9	$+0.20$	10.38
Terminal anchor	$1/16$	$1/16$	0.544	78.3	0.755	85.5	$+0.12$	2.64

Table 1: Selected operating points on both bases, with the Scale-RAE reward/cost audit.

q

is nominal post-lock external-score coverage and

q_{\mathrm{eff}}

its realized value; “External panel

\Delta

” reports the mean change from

\theta_{0}

under the fixed three-scorer anchor composite. GenEval and DPG-Bench are the held-out utilities. Timing excludes evaluation. Bold marks the unanchored failure, primary steelman, and full method.

Figure 5: Final utility versus external decode coverage on Scale-RAE. All anchored conditions use the same three-member external panel; the star marks the grounding steelman at $q=1$ . Non-inferiority follows the paired $95\%$ CIs in Eq. ( 12 ).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Question	Prespecified readout
Self-verifier capture	Self-reward rises while the parameter-disjoint external panel falls, with within-group diversity retained as the disambiguator against collapse.
Exploiting a fixed judge	Official GenEval decline in the frozen-semantic cell, supported by retained diversity and frozen-reference typicality.
Judge rewriting	Seed-level separation between shared and frozen self–external divergence values at matched rank, with fixed-probe AUC corruption as the functional confirmation.
Staleness	Spearman correlation of the frozen read with the external panel on current-policy samples, read against the offline drift axis.
Refresh behavior	Single-refresh and iterated self-refresh horizons, tracking roundwise recovery peaks and within-round decay.

Appendix

Table 2: Readout-to-question map for the diagnostic study. Operators are defined in Appendix B ; values are reported in Appendix E .

Seed	Semantic shared	Semantic frozen
1	6.24	2.99
2	6.51	2.31
3	6.03	1.93
4	6.48	2.31
5	4.85	2.54
Range	$4.85$ – $6.51$	$1.93$ – $2.99$

Appendix

Table 3: Per-seed self–external divergence for the semantic cells at rank

16

. All five shared values exceed all five frozen values, giving the exact one-sided probability

1/252

under exchangeability; the median gap clears the effect-size floor by

4.6\times

Cell	Typicality ( $<0.414$ )	Diversity ( $\theta_{0}{=}1$ )
Semantic shared	$0.378$	$0.846$
Semantic frozen	$0.345$	$0.907$
Denoising shared	$0.245$	$0.564$
Denoising frozen	$0.267$	$0.605$

Appendix

Table 4: Final typicality distance and within-group diversity per cell. All four cells stay on-manifold (distance below the

0.414

cutoff). The semantic cells retain diversity (

\approx 0.85

–

0.91

); the denoising cells collapse it (

\approx 0.56

–

0.61

Arm	GenEval	DPG	SE (GE / DPG)	$q$
Pretrained $\theta_{0}$	0.495	76.9	$1.35\,/\,0.58^{\dagger}$	—
Terminal-only warmed lock $q{=}0$	0.449	74.5	$1.00\,/\,0.42$	$0$
External panel $q{=}1$	0.515	77.8	$0.61\,/\,0.39$	$1$
Grounding steelman $q{=}1$	0.560	79.0	$0.70\,/\,0.35$	$1$
Terminal anchored $q{=}1/4$	0.531	78.0	$0.93\,/\,0.41$	$1/4$
Terminal anchored $q{=}1/16$	0.544	78.3	$1.13\,/\,0.28$	$1/16$

Appendix

Table 5: Scale-RAE per-arm external utility. SE is the training-seed standard error (GenEval points / DPG);

q

is external-decode coverage. Diagnostic cells and cost are reported in Appendices E.1 and G.6 .

Arm	GenEval	DPG
Pretrained $\theta_{0}$	0.730	85.0
Terminal-only warmed lock $q{=}0$	0.694	82.1
External panel $q{=}1$	0.743	85.2
Grounding steelman $q{=}1$	0.768	85.9
Terminal anchored $q{=}1/4$	0.746	85.3
Terminal anchored $q{=}1/16$	0.755	85.5

Appendix

Table 6: Show-o2-1.5B per-arm external utility.

Base / metric	$\tfrac{1}{2}$ active	$2\,\mathrm{SE}(\theta_{0})$	$\delta_{\mathrm{NI}}$	Threshold
Scale-RAE GenEval	5.55	2.70	5.55	50.45
Scale-RAE DPG	2.25	1.16	2.25	76.75
Show-o2 GenEval	3.70	2.30	3.70	73.10
Show-o2 DPG	1.90	0.96	1.90	84.00

Appendix

Table 7: Non-inferiority margins and utility thresholds. GenEval rows in points (

1\text{ pt}=0.01

); DPG rows in native units. The active-control term binds in every row, so the floor never sets the margin.

Arm	$\Delta$ (GE / DPG)	SE (GE / DPG)	95% lower (GE / DPG)	Joint decision
Terminal anchored $q{=}1/4$	$-2.9\,/\,-1.0$	$0.80\,/\,0.34$	$-5.12\,/\,-1.94$	pass
Terminal anchored $q{=}1/16$	$-1.6\,/\,-0.7$	$0.85\,/\,0.36$	$-3.96\,/\,-1.70$	pass
Dense fixed-dose $q{=}1/16$	$+2.5\,/\,+1.5$	$0.78\,/\,0.33$	$+0.33\,/\,+0.58$	pass (both ${>}0$ )
Dense fixed-dose $q{=}1/32$	$-0.4\,/\,+0.3$	$0.80\,/\,0.35$	$-2.62\,/\,-0.67$	pass
Dense fixed-dose $q{=}1/64$	$-5.9\,/\,-3.2$	$0.78\,/\,0.33$	$-8.07\,/\,-4.12$	fail (both)
Dense controller $q{=}1/16$	$+4.2\,/\,+2.3$	$0.72\,/\,0.30$	$+2.20\,/\,+1.47$	pass (both ${>}0$ )

Appendix

Table 8: Scale-RAE non-inferiority decisions (arm minus the grounding steelman). Pass iff both lower bounds

\geq-5.55/-2.25

Arm	$\Delta$ (GE / DPG)	SE (GE / DPG)	95% lower (GE / DPG)	Joint decision
Terminal anchored $q{=}1/4$	$-2.2\,/\,-0.6$	$0.42\,/\,0.30$	$-3.37\,/\,-1.43$	pass
Terminal anchored $q{=}1/16$	$-1.3\,/\,-0.4$	$0.65\,/\,0.25$	$-3.10\,/\,-1.09$	pass
Dense fixed-dose $q{=}1/16$	$+1.2\,/\,+0.3$	$0.58\,/\,0.24$	$-0.41\,/\,-0.37$	pass
Dense fixed-dose $q{=}1/32$	$-0.7\,/\,-0.2$	$0.58\,/\,0.24$	$-2.31\,/\,-0.87$	pass
Dense fixed-dose $q{=}1/64$	$-4.2\,/\,-1.6$	$0.58\,/\,0.22$	$-5.81\,/\,-2.21$	fail (both)
Dense controller $q{=}1/16$	$+2.4\,/\,+0.6$	$0.55\,/\,0.20$	$+0.87\,/\,+0.04$	pass (both ${>}0$ )

Appendix

Table 9: Show-o2 non-inferiority decisions (arm minus the grounding steelman). Pass iff both lower bounds

\geq-3.70/-1.90

Reward read	Divergence gap	Probe-AUC loss	Panel change	Diversity
Shared	$+3.42$	0.18	$-1.19$	0.83
EMA ( $\beta{=}0.99$ )	$+1.96$	0.12	$-0.83$	0.89
Warmed-then-locked ( $W{=}250$ )	$+0.37$	0.056	$-0.54$	0.87
Frozen	$-0.01$	0	$-0.58$	0.90

Appendix

Table 10: Structural intervention ladder (Scale-RAE diagnostic instruments, five-seed medians). The divergence gap and probe-AUC loss track the rewriting route; the external-panel change (panel-

z

margin) tracks the exploitation residual. Divergence and panel-change seed ranges: shared

[2.84,4.18]/[-1.28,-1.14]

, EMA

[1.61,2.03]/[-0.87,-0.71]

, warmed-locked

[0.32,0.47]/[-0.63,-0.52]

, frozen

[-0.03,0.02]/[-0.61,-0.51]

$q$	Locked	Shared
$0$	$-0.54$	$-1.19$
$1/64$	$-0.36$	$-0.98$
$1/16$	$-0.03$	$-0.59$
$1/4$	$+0.14$	$-0.27$

Appendix

Table 11: Raw median external-panel change under locked and shared reward reads. Retention is the one-sided region in which the panel change is

\geq-\delta_{\mathrm{panel}}=-0.17

$\beta$	$\Delta$ self	Panel change	Learn	Ext	Both
$0$	$+3.89$	$-1.19$	5/5	0/5	0/5
$0.1$	$+3.41$	$-0.90$	5/5	0/5	0/5
$0.3$	$+2.39$	$-0.64$	5/5	0/5	0/5
$0.5$	$+1.76$	$-0.48$	2/5	0/5	0/5
$0.7$	$+1.41$	$-0.38$	0/5	0/5	0/5
$0.9$	$+0.97$	$-0.16$	0/5	3/5	0/5

Appendix

Table 12: Dense KL/proximity sweep (five seeds per

\beta

). “Learn” is seeds meeting self-reward gain

\geq 1.9

; “Ext” is seeds whose panel change is inside the retention tolerance (

\geq-\delta_{\mathrm{panel}}=-0.17

); “Both” is seeds meeting both. The two gates are satisfied in disjoint

\beta

regions and never jointly.

Coverage $q$	GenEval	DPG
$0$	0.449	74.5
$1/16$	0.544	78.3
$1/4$	0.531	78.0
$1$ (external panel)	0.515	77.8

Appendix

Table 13: Scale-RAE terminal-only anchoring frontier using the external panel after decoding. Both metrics peak at

q=1/16

$q$	Controller	Fixed-dose	Gap	CI $>0$ on both
	(GE / DPG)	(GE / DPG)	(GE / DPG)
Scale-RAE
$1/4$	$0.584\,/\,80.1$	$0.582\,/\,80.0$	$+0.2\,/\,+0.1$	no
$1/16$	$0.602\,/\,81.3$	$0.585\,/\,80.5$	$+1.7\,/\,+0.8$	yes
$1/32$	$0.575\,/\,80.1$	$0.556\,/\,79.3$	$+1.9\,/\,+0.8$	yes
$1/64$	$0.535\,/\,78.0$	$0.501\,/\,75.8$	$+3.4\,/\,+2.2$	yes

Appendix

Table 14: Controller versus dense fixed-dose at matched coverage. The paired gap is controller minus fixed-dose (GenEval points / DPG). “CI

>0

on both” records whether the paired

95\%

intervals exclude zero on both metrics.

Paired contrast	$\Delta$ (GE / DPG)	CI $>0$ on both
Active vs. random selection	$+1.3\,/\,+0.65$	yes
Residual propagation vs. gate-and-replace	$+1.0\,/\,+0.45$	yes
Drift triggering vs. fixed scheduling	$+0.9\,/\,+0.40$	yes
Full controller vs. fixed-dose backbone	$+1.7\,/\,+0.80$	yes

Appendix

Table 15: Matched-budget component ablations at

q=1/16

(Scale-RAE).

\Delta

is the seed-paired gain in GenEval points / DPG. All four paired

95\%

intervals exclude zero on both metrics.

Arm	$q$	$q_{\mathrm{eff}}$	Panel $\Delta$	s / step
Terminal-only warmed lock	$0$	$0$	$-0.52$	2.51
External panel	$1$	$1$	$+1.39$	7.43
Grounding steelman	$1$	$1$	$+0.20$	10.38
Terminal anchored	$1/16$	$1/16$	$+0.12$	2.64
Dense fixed-dose	$1/16$	$1/16$	$+0.29$	2.88
Dense fixed-dose	$1/32$	$1/32$	$+0.14$	2.70

Appendix

Table 16: Reward and cost audit on Scale-RAE.

q

is nominal external-decode coverage and

q_{\mathrm{eff}}

its realized value; “Panel

\Delta

” is the mean change from

\theta_{0}

on the fixed three-scorer anchor composite; timing excludes evaluation.

Item	Configuration
Primary base (Scale-RAE)	Qwen2.5-1.5B language model with a $2.4$ B DiT in a $1152$ -dimensional SigLIP-2 latent space; $\approx 3.9$ B core-model parameters.
Second base (Show-o2)	$\approx 1.5$ B unified model, shared Qwen backbone, 3D causal-VAE latent space, flow matching; used for cross-architecture validation of unanchored capture and method transfer.
Decoder	Each base’s frozen decoder (RAE for Scale-RAE and 3D-VAE for Show-o2), invoked only for external scoring or evaluation, never by a latent self-verifier.
Trainable adapters	Capacity-matched language-model and DiT trunk adapters $\phi$ ; all primary diagnostic cells use matched rank $16$ .
Frozen, out of optimizer	Understanding-side projector $\omega$ (no adapter); trunk base weights $\theta_{\mathrm{sh}}^{\mathrm{base}}$ .
Seeds	$5$ training seeds per reported arm; $3$ for the iterated-refresh diagnostic.

Appendix

Table 17: Common training configuration and the iterated-refresh extension.

Item	Configuration
Optimizer	AdamW; learning rate $10^{-4}$ ; constant schedule with no learning-rate warm-up.
AdamW and clipping	Weight decay $0.01$ ; betas $(0.9,0.999)$ ; optimizer epsilon $10^{-8}$ ; maximum gradient norm $1.0$ .
Effective batch	One prompt group of $G{=}12$ rollouts per optimizer step; gradient accumulation $1$ and one PPO epoch.
TVE–Gibbs policy	Trajectory energy summed over all $50$ Euler solver steps; Gibbs temperature $\tau{=}1.0$ .
PPO and proximity	Symmetric clip $\epsilon{=}0.2$ ; $\beta{=}0$ in the primary diagnostic and repaired arms, with nonzero $\beta$ used only in the explicit KL sweep.

Appendix

Table 18: Optimization and policy-update configuration.

Prompt-pool audit	Count
One-caption-per-image pool	$118{,}287$
Normalized unique captions	$117{,}574$
Exact matches (GenEval / DPG-Bench)	$0/0$
Near-verbatim removals (GenEval / DPG-Bench)	$186/5$
Final frozen pool	$117{,}383$

Appendix

Table 19: Training-prompt construction and benchmark-overlap audit.

Block	$w$	$95\%$ CI	Majority	$p$	$\alpha_{\mathrm{K}}$
Q1: prompt faithfulness
P1 ( $\theta_{0}$ )	0.679	$[0.588,0.765]$	$70/24$	$6.8\mathrm{e}{-}6$	0.52
P2 ( $\theta_{0}$ )	0.741	$[0.657,0.822]$	$78/18$	$5.4\mathrm{e}{-}9$	0.58
P3 (controller)	0.611	$[0.517,0.703]$	$58/34$	$0.016$	0.44
Q2: visual quality
P1 ( $\theta_{0}$ )	0.540	$[0.470,0.610]$	—	—	0.28

Appendix

Table 20: Blind paired human win rates on Scale-RAE.

w

is the decided-pair win rate for the first-named arm of each block, with its two-stage bootstrap

95\%

interval; “Majority” counts prompts whose rater majority favors each arm;

p

is the Holm-adjusted two-sided exact binomial test on those counts.

	P1 (frozen)		P2 (shared)
Tag	$n$	Share	$n$	Share
Action or pose	68	0.407	83	0.439
Spatial relation	55	0.329	68	0.360
Scene or context	48	0.287	59	0.312
Color or attribute	32	0.192	40	0.212
Count	23	0.138	28	0.148

Appendix

Table 21: Failure-mode tags on judgments where the post-RL image was judged less faithful. Shares are of

167

(P1) and

189

(P2) judgments and sum above one because the tag is multi-select.

Response	P1 (frozen)	P2 (shared)
Neither image	262	251
$\theta_{0}$ only	14	15
Post-RL only	19	27
Both	5	7
Neither rate	0.873	0.837
$\theta_{0}$ flag rate	0.063	0.073

Appendix

Table 22: Visible-artifact responses over the

300

judgments of each block. Flag rates count every judgment in which an arm is named, alone or together with the other.

Instrument	P1	P2	P3	Pooled
External panel	0.777	0.823	0.674	0.759
decided prompts	94	96	92	282
Spearman (panel)	0.52	0.57	0.38	0.49
GenEval	0.710	0.758	0.621	0.699
DPG-Bench	0.766	0.813	0.652	0.745

Appendix

Table 23: Agreement of the automated instruments with the human majority. The Spearman row correlates the prompt-level external-panel margin with the prompt-level human win rate. GenEval and DPG-Bench rows use only their own

50

prompts.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Apr 13, 2026Haozhe Wang, Cong Wei, Weiming Ren +3Reward Modeling Multimodal Reward Modeling
HKUST · University of Waterloo · Alibaba
SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
Oct 14, 2025Weiyang Jin, Yuwei Niu, Jiaqi Liao +4Unified Multimodal Models T2I Generation
HKU MMLab · The University of Hong Kong · Shenzhen Loop Area Institute +2
Visual Jev Rewards: Reference-Bound Verification for Multi-Subject Image Generation
Oct 7, 2026Baoteng Li, Wenzhuo Wu, Kongming Liang +1Multi-Subject Image Generation Reward Modeling
School of Artificial Intelligence, Beijing University of Posts and Telecommunications1 · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance2

Apr 13, 2026J/K move · Enter open · S save

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

Haozhe Wang, Cong Wei, Weiming Ren, Jiaming Liu+2

HKUST · University of Waterloo · Alibaba

Reward Modeling Multimodal Reward Modeling RL for Image Generation Prompt Optimization for T2I Generation

Open paper AskPDF arXiv

Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.

Abstract

Reward Hacking

Explore similar work

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

Visual Jev Rewards: Reference-Bound Verification for Multi-Subject Image Generation