Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models

Lit Sin Tan; Junzhe Chen; Xiaolong Fu; Lichen Ma; Junshi Huang; Jianzhong Shi; Yan Li; Lijie Wen

cs.LGMar 16, 2026

Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models

Authors: Lit Sin Tan, Junzhe Chen, Xiaolong Fu, Lichen Ma, Junshi Huang, Jianzhong Shi, Yan Li, Lijie Wen

Organizations: Tsinghua University

Abstract

Existing test-time scaling (TTS) methods for unified multimodal models (UMMs) in text-to-image (T2I) generation primarily rely on search or sampling strategies that produce only instance-level improvements, limiting the ability to learn from prior inferences and accumulate knowledge across similar prompts. To overcome these limitations, we propose Meta-TTRL, a metacognitive test-time reinforcement learning framework. Meta-TTRL performs test-time parameter optimization guided by model-intrinsic monitoring signals derived from the meta-knowledge of UMMs, achieving self-improvement and capability-level improvement at test time. Extensive experiments demonstrate that Meta-TTRL generalizes well across three representative UMMs, including Janus-Pro-7B, BAGEL, and Qwen-Image, achieving significant gains on compositional reasoning tasks and multiple T2I benchmarks with limited data. We provide the first comprehensive analysis to investigate the potential of test-time reinforcement learning (TTRL) for T2I generation in UMMs. Our analysis further reveals a key insight underlying effective TTRL: metacognitive synergy, where monitoring signals align with the model's optimization regime to enable self-improvement.

Figures & tables

Figure 1: Test-Time Reinforcement Learning in UMMs.

Figure 2: Meta-TTRL operates with a meta-level introspector that constructs rubric and provides intrinsic monitoring signals, and an object-level generator that produces candidate images. These intrinsic signals are aggregated as rewards to guide policy optimization and improve T2I generation. For clarity, the figure omits the question index $m$ and shows only representative verification questions.

Model	TIIF-Bench $\uparrow$		T2I-CompBench++ $\uparrow$								DPG $\uparrow$
Model	Short	Long	Color	Shape	Texture	2D Spatial	3D Spatial	Non- Spatial	Numeracy	Complex	Score
Qwen-Image	83.45	82.48	82.80	59.86	74.51	44.30	45.36	31.71	75.82	38.38	88.32
w/ Meta-TTRL	85.28	84.67	84.50	62.42	76.36	46.59	46.11	31.85	76.77	38.48	89.00
$\triangle$	+1.83	+2.19	+1.70	+2.56	+1.85	+2.29	+0.75	+0.14	+0.95	+0.10	+0.68
	$\uparrow 2.19\%$	$\uparrow 2.66\%$	$\uparrow 2.05\%$	$\uparrow 4.28\%$	$\uparrow 2.48\%$	$\uparrow 5.17\%$	$\uparrow 1.65\%$	$\uparrow 0.44\%$	$\uparrow 1.25\%$	$\uparrow 0.26\%$	$\uparrow 0.77\%$
BAGEL	71.65	70.32	80.81	57.94	71.54	34.03	37.78	31.11	65.52	37.78	84.03

Table 1: Evaluation results on TIIF-Bench ( Wei et al., 2025 ) , T2I-CompBench++ ( Huang et al., 2025 ) , DPG-Bench ( Hu et al., 2024 ) . Arrows (

\uparrow

) denote that higher is better. The T2I-CompBench++ score is normalized to a 0–100 scale for visualization.

Figure 3: Qualitative case studies of Meta-TTRL

Model	Train on	Vanilla	Meta-TTRL
Janus-Pro-7B	T2I++	77.80	81.14 +3.34
	TIIF		79.40 +1.60
	DPG		78.45 +0.65
Bagel	T2I++	79.24	83.37 +4.13
	TIIF		82.67 +3.43
	DPG		82.67 +3.43

Table 2: Performance on GenEval for all evaluated models

Train on	Eval on	Vanilla	Meta-TTRL
T2I++	TIIF	65.68	68.71 +3.03
T2I++	DPG	83.84	84.63 +0.79
TIIF	T2I++	35.25	40.44 +5.19
TIIF	DPG	83.84	84.32 +0.48
DPG	T2I++	35.25	38.93 +3.68
DPG	TIIF	65.68	66.71 +1.03

Table 3: Cross-benchmark generalization on Janus-Pro-7B.

Panel of Figure 4 — Figure 4: Performance comparison between Meta-TTRL (blue) and E-TTRL (red) on T2I-CompBench++ across eight subdimensions.

Figure 5: Performance comparison of Baseline, RL Leakage with UnifiedReward, and Meta-TTRL across three T2I benchmarks. Left: Radar plot comparing performance on T2I-CompBench++. Right: Bar charts showing results on TIIF-Bench and DPG-Bench.

Train on	Janus-Pro-7B			Bagel			Qwem-Image
	Vanilla	GIT	Meta-TTRL	Vanilla	GIT	Meta-TTRL	Vanilla	GIT	Meta-TTRL
T2I++	35.25	47.16 +11.91	48.90 +13.65	52.60	54.83 +2.23	55.13 +2.53	56.71	57.71 +1.00	57.88 +1.17
TIIF	65.68	69.73 +4.05	71.08 +5.40	71.00	72.70 +1.70	75.29 +4.29	82.97	84.24 +1.27	84.98 +2.01
DPG	83.84	84.78 +0.94	85.67 +1.83	84.03	86.24 +2.21	86.33 +2.30	88.32	88.83 +0.51	89.00 +0.68

Table 4: Comparison between Meta-TTRL and replacing rubric evaluation with GIT across three benchmarks.

Figure 6: Training Dynamics of Meta-TTRL

Test-Time RL

Test-Time Reinforcement Learning adapts policies at deployment using interaction or self-generated feedback, without a separate task-specific training phase.

See all 48 papers

SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models
Oct 14, 2025Weiyang Jin, Yuwei Niu, Jiaqi Liao +4Unified Multimodal Models T2I Generation
HKU MMLab · The University of Hong Kong · Shenzhen Loop Area Institute +2
Recursive Self-Improvement in Unified Multimodal Models
Oct 2, 2026Huijuan Wang, Chufan Shi, Cheng Yang +3Unified Multimodal Models Chart Understanding
University of Southern California · University of California San Diego · Carnegie Mellon University
Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models
Mar 6, 2026Jiadong Pan, Liang Li, Yuxin Peng +6Unified Multimodal Models T2I Generation
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Peking University +2

Oct 14, 2025J/K move · Enter open · S save

SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

Weiyang Jin, Yuwei Niu, Jiaqi Liao, Chengqi Duan+3

HKU MMLab · The University of Hong Kong · Shenzhen Loop Area Institute+2

Unified Multimodal Models T2I Generation Generative Modeling Multimodal Reward Modeling RL for Image Generation Multimodal Foundation Models

Open paper AskPDF arXiv

Recently, remarkable progress has been made in Unified Multimodal Models (UMMs), which integrate vision-language generation and understanding capabilities within a single framework. However, a model's strong visual understanding often fails to transfer to visual generation: it may correctly judge prompt-image alignment while failing to generate a faithful image from the same prompt. This raises a compelling question: Can a model improve itself by using its understanding module to reward its generation module? We introduce SRUM, a self-rewarding post-training framework directly applicable to existing UMMs of various designs. SRUM creates a feedback loop where the model's own understanding module acts as an internal ``evaluator'', providing corrective signals to improve generation without additional human-labeled data or external reward models. To provide comprehensive feedback, SRUM uses a global-local dual reward system: a global reward ensures overall visual semantics and layout, while a local reward refines fine-grained, object-level fidelity. SRUM shows strong generalization, boosting performance on T2I-CompBench from 82.18 to 88.37 and on T2I-ReasonBench from 43.82 to 46.75. Overall, our work establishes a powerful paradigm for enabling a UMM's understanding module to guide and enhance its own generation via self-rewarding.

Abstract

Test-Time RL

Explore similar work

SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

Recursive Self-Improvement in Unified Multimodal Models

Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models