OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generation

Bingyang Cui; Yujie Zhang; Yiling Xu; Yunfeng Guan

cs.CVOct 2, 2026

OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generation

Authors: Bingyang Cui, Yujie Zhang, Yiling Xu, Yunfeng Guan

Organizations: Shanghai Jiao Tong University

Abstract

Reinforcement learning (RL) for Text-to-3D (T23D) generation requires optimization across multiple quality dimensions such as semantic alignment and texture clarity. Existing methods typically optimize these dimensions simultaneously through multiple reward aggregation, without explicitly modeling inter-dimension dependencies. This can cause imbalanced optimization and persistent interference among conflicting dimensions. To address this limitation, we propose OuroReward, an interference-aware sequential reward scheduling strategy for T23D RL. OuroReward first estimates pairwise dependencies among dimensions and constructs a cyclic optimization path that minimizes cumulative interference. By incorporating the tail-to-head dependency, the cycle captures global compatibility across the entire schedule. Then, OuroReward converts the cycle into a one-pass sequence, and starts optimization from the dimension with the lowest aggregate interference. Rather than assigning a fixed optimization budget to each dimension-wise reward, training adaptively determines when to advance to the next reward according to the remaining optimization headroom of the current one. We further introduce AdaSelect, an adaptive prompt selection strategy that identifies reliable and informative prompts aligned with the model's current capability. By focusing policy updates on these prompts, AdaSelect effectively improves training stability. Extensive experiments across different T23D models and RL algorithms demonstrate that our framework consistently improves generation quality across multiple dimensions.

Figures & tables

Panel of Figure 1 — Fig. 1 : (a) Representative failure cases produced by different T23D generation models. (b) Comparison of gradient discrepancy and reward-change conflict ratio between a conflicting dimension pair ( G $\leftrightarrow$ T ) and a compatible dimension pair ( A $\leftrightarrow$ O ) under the default reward-averaging strategy. A , G , T , and O denote four quality dimensions including Alignment , Geometry , Texture , and Overall , respectively.

Fig. 2 : The framework of the proposed AdaSelect strategy.

Fig. 3 : Comparison of Random and AdaSelect for Prompt Selection at Step $t$ .

Fig. 4 : The framework of the proposed OuroReward strategy and the RL training policy.

Panel of Figure 5 — Fig. 5 : Statistical analysis of the constructed large-scale prompt dataset. (a) Prompt composition, where the inner and outer circles represent interaction types and attribute categories, respectively. (b) Word cloud visualization of the prompt corpus. (c) Distribution of prompt lengths.

Model	Trained Tasks		Unseen Tasks
Model	HyperScore	Rank2Score	CLIPScore	ImageReward	Uni3D	Qwen3-VL-8B- Instruct	Win Rate
TRELLIS	3.466	3.713	0.208	0.362	0.254	6.483	-
HyperScore Task
+ GRPO	3.782	3.696	0.205	0.364	0.252	6.479	0.007
+ DiffusionNFT	4.050	3.798	0.213	0.467	0.261	6.671	0.033
+ GRPO + Ours	4.278	3.861	0.219	0.598	0.260	6.710	0.337

TABLE I : Performance comparison of our method with representative baselines across proxy rewards and unseen evaluators. Our method comprises two components: AdaSelect and OuroReward. The proxy reward is marked in orange. The best result in each column is shown in bold , while the second is underlined .

Panel of Figure 7 — Fig. 6 : Learning curves and dimension-wise proxy reward performance under different base models. (a) TRELLIS-base as the base model. (b) Hunyuan3D-2.1 as the base model.

Fig. 7 : Qualitative results generated by the proposed method and base RL algorithms across different prompts.

Base Model	Aggregation Strategy	Step	A $\leftrightarrow$ G	A $\leftrightarrow$ T	A $\leftrightarrow$ O	G $\leftrightarrow$ T	G $\leftrightarrow$ O	T $\leftrightarrow$ O	Mean Gradient Similarity	Mean Reward PLCC
TRELLIS	Average	100	0.361	0.384	0.734	0.310	0.595	0.697	0.514	0.817
	FocalReward	100	0.407	0.438	0.716	0.391	0.629	0.681	0.544	0.837
	OuroReward	100	0.513	0.527	0.741	0.484	0.692	0.723	0.613	0.865
	Average	150	0.416	0.425	0.749	0.393	0.631	0.718	0.555	0.824
	FocalReward	150	0.441	0.470	0.724	0.443	0.662	0.706	0.574	0.849
	OuroReward	150	0.562	0.575	0.758	0.537	0.725	0.734	0.649	0.873

TABLE II : Comparison of inter-dimension gradient similarity and reward PLCC across training steps under different training strategies. Rank2Score as the proxy reward model and DiffusionNFT as the RL algorithm. The best result in each column is shown in bold , while the second is underlined .

Base Model	Aggregation Strategy	Rank2 Score	CLIP Score	Image Reward	Uni3D
TRELLIS	Average	4.374	0.221	0.567	0.258
	FocalReward	4.410	0.224	0.582	0.259
	OuroReward	4.525	0.233	0.678	0.267
Hunyuan3D-2.1	Average	4.428	0.239	0.651	0.271
	FocalReward	4.457	0.240	0.684	0.270
	OuroReward	4.508	0.248	0.783	0.278

TABLE III : Comparison of multi-dimensional aggregation strategy. The proxy reward is marked in orange. The best result in each column is shown in bold , while the second is underlined .

Fig. 8 : Illustration of dimension-inconsistent failure cases under different RL optimization strategies. Rank2Score as the proxy reward model and DiffusionNFT as the RL algorithm.

Ada- Select	Ouro- Reward	Rank2Score	CLIPScore	Image Reward	Uni3D
✗	✗	4.253	0.216	0.483	0.253
✓	✗	4.374	0.221	0.567	0.258
✗	✓	4.496	0.230	0.635	0.262
✓	✓	4.525	0.233	0.678	0.267

TABLE IV : Ablation on AdaSelect and OuroReward. The proxy reward is marked in orange. The best result in each column is shown in bold , while the second is underlined .

Gradient Similarity	Reward PLCC	Rank2Score	CLIPScore	Image Reward	Uni3D
✗	✗	4.374	0.221	0.567	0.258
✓	✗	4.482	0.228	0.629	0.264
✗	✓	4.435	0.224	0.596	0.260
✓	✓	4.525	0.233	0.678	0.267

TABLE V : Ablation on dependency score criteria. The proxy reward is marked in orange. The best result in each column is shown in bold , while the second is underlined .

Type	Path Order	Rank2Score	CLIP Score	Image Reward	Uni3D
Baseline	Average	4.374	0.221	0.567	0.258
Random	O $\to$ G $\to$ A $\to$ T	4.435	0.224	0.596	0.260
Random	O $\to$ A $\to$ T $\to$ G	4.482	0.228	0.629	0.264
Reversed	T $\to$ G $\to$ A $\to$ O	4.361	0.218	0.501	0.253
Shifted	A $\to$ G $\to$ T $\to$ O	4.469	0.226	0.613	0.259
Ours	O $\to$ A $\to$ G $\to$ T	4.525	0.233	0.678	0.267

TABLE VI : Ablation on sequential optimization order. The proxy reward is marked in orange. The best result in each column is shown in bold , while the second is underlined .

Proxy Reward	Uni3D	Qwen3-VL- 8B-Instruct	Hyper Score	Rank2 Score
Uni3D	0.540	6.501	3.578	3.732
Qwen3-VL-8B-Instruct	0.263	7.851	3.637	3.870
HyperScore	0.266	6.835	4.483	3.937
Rank2Score	0.267	6.982	3.654	4.525

TABLE VII : Ablation on proxy reward model. The proxy reward is marked in orange. The best result in each column is shown in bold , while the second is underlined .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Metric	MATE-3D				T23D-CompBench				Time
Type	Metric	Alignment	Geometry	Texture	Overall	Alignment	Geometry	Texture	Overall	(s / sample)
Projection-based	CLIPScore [ 78 ]	0.494	0.496	0.537	0.510	0.552	0.501	0.536	0.562	$\sim 10^{-1}$
	ImageReward [ 74 ]	0.658	0.673	0.686	0.679	0.616	0.774	0.821	0.832	$\sim 10^{-1}$
	HyperScore [ 69 ]	0.739	0.782	0.811	0.792	0.728	0.828	0.889	0.860	$\sim 10^{-1}$
	Rank2Score [ 70 ]	0.868	0.898	0.894	0.897	0.907	0.888	0.932	0.934	$\sim 10^{-1}$
Model-based	ULIP-T (Unit) [ 79 ]	0.435	0.467	0.472	0.469	0.634	0.657	0.661	0.663	$\sim 10^{-1}$

Appendix

TABLE S8 : Performance comparison for different reward models on MATE-3D and T23D-CompBench benchmarks. The top two performance results are marked in boldface and underline .

Fig. S9 : Qualitative results generated by the proposed method and base RL algorithms across different prompts.

Fig. S10 : The framework of the proposed Adaptive Curriculum Prompt Selection Strategy.

Prompt Selection for RL

See all 9 papers

Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference
Mar 2, 2026Jiaqi Leng, Shuyuan Tu, Haidong Cao +4Text-to-3D Generation Diffusion Model Alignment
Fudan University
HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilities
Jun 12, 2026Yijun Liu, Jie Huang, Zeyue Xue +5Diffusion Model Alignment Multimodal Reward Modeling
Tsinghua University · JD Explore Academy · Peking University +1
ReConText3D: Replay-based Continual Text-to-3D Generation
Apr 15, 2026Muhammad Ahmed Ullah Khan, Muhammad Haris Bin Amir, Didier Stricker +1Text-to-3D Generation 3D Asset Generation
DFKI · RPTU Kaiserslautern-Landau

Mar 2, 2026J/K move · Enter open · S save

Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference

Jiaqi Leng, Shuyuan Tu, Haidong Cao, Sicheng Xie+3

Fudan University

Text-to-3D Generation Diffusion Model Alignment Diffusion Model Guidance Score Distillation Sampling

Open paper AskPDF arXiv

Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data-scarce 3D domains. To address this, we propose Preference Score Distillation (PSD), an optimization-based framework that leverages pretrained 2D reward models for human-aligned text-to-3D synthesis without 3D training data. Our key insight stems from the incompatibility of pixel-level gradients: due to the absence of noisy samples during reward model training, direct application of 2D reward gradients disturbs the denoising process. Noticing that similar issue occurs in the naive classifier guidance in conditioned diffusion models, we fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through our implicit reward model. Furthermore, recognizing that frozen pretrained diffusion models constrain performance, we introduce an adaptive strategy to co-optimize preference scores and negative text embeddings. By incorporating CFG during optimization, online refinement of negative text embeddings dynamically enhances alignment. To our knowledge, we are the first to bridge human preference alignment with CFG theory under score distillation framework. Experiments demonstrate the superiority of PSD in aesthetic metrics, seamless integration with diverse pipelines, and strong extensibility.

Abstract

Prompt Selection for RL

Explore similar work

Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference

HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilities

ReConText3D: Replay-based Continual Text-to-3D Generation