MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design

Jixin Tang; Ji Guo; Jun Zhao

cs.LGOct 9, 2026

MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design

Authors: Jixin Tang, Ji Guo, Jun Zhao

Organizations: Fudan University

Abstract

We introduce MORAD, an online reinforcement learning framework for backbone-conditioned RNA diffusion models that trains a single shared policy across diverse target backbones. Structure-conditioned RNA inverse design seeks nucleotide sequences compatible with a target three-dimensional backbone, yet existing reinforcement learning approaches either optimize a separate policy for each target backbone or rely on offline preference optimization, leaving online post-training of a reusable backbone-conditioned model largely unexplored. MORAD combines backbone-conditioned group sampling, group-relative reward normalization, and a forward-process diffusion objective. Beyond existing RL rewards that primarily target tertiary structural quality, MORAD jointly optimizes six measurements spanning tertiary geometry, base pairing, ensemble behavior, and sequence composition. To aggregate these heterogeneous objectives despite differences in scale and optimization direction, MORAD maps each measurement to a bounded desirability score and combines them with a weighted geometric mean. Compared with a tertiary-structure-only reward, this multi-objective formulation improves 10 of the 11 evaluation metrics that have a preferred direction. Applied to pretrained RIDE, MORAD improves GDT-TS by 16.0% (from 0.3300 to 0.3827) and pairing MCC by 19.4% (from 0.6110 to 0.7295), while reducing normalized ensemble defect by 27.5% (from 0.3906 to 0.2830). On the test set, MORAD outperforms RiboDiffusion, gRNAde, and RDesign across tertiary structural accuracy, secondary-structure pairing, and thermodynamic quality, establishing a new state of the art in structure-conditioned RNA inverse design.

Figures & tables

Figure 1: MORAD training loop. Backbone-conditioned sampling produces candidate sequences. Three evaluation channels provide the six-component reward and group-relative weights. Forward-noised candidates train the shared policy under a frozen-reference constraint; EMA feedback refreshes the sampler. The update follows the positive/negative objective in Eq. ( 3 ); the lower feedback path denotes EMA refresh.

Figure 2: RIDER and MORAD reward construction. RIDER scores tertiary structural consistency ( Hu et al., 2026 ) . MORAD combines three tertiary measurements, pairing agreement, normalized ensemble defect (ED), and composition deviation. Each desirability score lies in

[\delta,1]

;

\phi_{j}

denotes the map in Eq. ( 4 ), and

\widetilde{w}_{j}

is the normalized weight in Eq. ( 5 ). The panels compare reward definitions.

Model	GDT-TS $\uparrow$	TM(C4 $\prime$ ) $\uparrow$	RMSD (Å) $\downarrow$	TM(C1 $\prime$ ) $\uparrow$	Success $\uparrow$	Recovery $\uparrow$	lDDT(C4 $\prime$ ) $\uparrow$	pLDDT $\uparrow$	Coarse clash $\downarrow$
RiboDiffusion	0.3132	0.2839	10.9215	0.3613	0.2549	0.5165	0.6174	0.6937	73.5383
gRNAde ( $T=0.1$ )	0.3591	0.3179	8.8374	0.3642	0.2484	0.5115	0.6607	0.6482	23.7261
RDesign	0.3291	0.2902	10.3441	0.3392	0.2549	0.4720	0.6368	0.6202	40.4457
RIDE	0.3300	0.2841	9.7199	0.3538	0.2484	0.5039	0.6622	0.6671	22.7503
MORAD	0.3827	0.3400	7.9196	0.3855	0.3007	0.5056	0.6774	0.6726	11.9438

Table 1: Tertiary-structure evaluation. First sampled candidate (candidate 0) for each of the 153 evaluation targets; C1

\prime

TM has 126 valid targets. MORAD is the checkpoint at update 390. Alignment scores and additional diagnostics are shown together; RMSD is in Å. Bold marks the best mean. Metric definitions are in Appendix D.2 .

Model	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	MCC $\uparrow$
RiboDiffusion	0.3563	0.3074	0.3212	0.3267
gRNAde ( $T=0.1$ )	0.7276	0.6099	0.6482	0.6734
RDesign	0.7084	0.6110	0.6462	0.6566
RIDE	0.6472	0.5717	0.5939	0.6110
MORAD	0.7824	0.6816	0.7226	0.7295

Table 2: Candidate-0 secondary-structure agreement. EternaFold predictions are compared with base pairs extracted from the native 3D structures ( Wayment-Steele et al., 2022 ) . Bold marks the best mean. Valid counts for each metric and model are in Appendix D.1.1 ; thermodynamic defect is reported separately in Table 3 .

Model	MFE $\sim$	EFE $\sim$	NED $\downarrow$	PE $\downarrow$	MFEfreq $\uparrow$
RiboDiffusion	-21.716	-22.847	0.6056	0.4284	0.2980
gRNAde ( $T=0.1$ )	-28.092	-28.918	0.3458	0.2652	0.4289
gRNAde ( $T=0.8$ )	-25.186	-26.212	0.3830	0.3260	0.3266
RDesign	-28.863	-29.708	0.3349	0.2775	0.3961
RIDE	-23.614	-24.643	0.3906	0.3059	0.3350
MORAD	-30.149	-30.937	0.2830	0.2102	0.4402

Table 3: Thermodynamic evaluation on 153 targets. MFE and EFE are minimum and ensemble free energies (kcal/mol); NED is normalized ensemble defect; PE is positional entropy; MFEfreq is the probability of the minimum-energy structure ( Lorenz et al., 2011 ; Zadeh et al., 2011 ) . MORAD is update 390. Bold marks the best mean;

\sim

marks metrics without a preferred direction, shown for context. Appendix D.3 reports all nine metrics and the native reference.

Tertiary structure
Measurement	RIDE	+ Structural reward	+ MORAD reward
GDT-TS $\uparrow$	0.3300	0.3627	0.3694
TM-score $\uparrow$	0.2841	0.3194	0.3358
RMSD (Å) $\downarrow$	9.720	8.369	8.207
C1 $\prime$ TM $\uparrow$	0.3538	0.3818	0.3919
Secondary structure and recovery

Table 4: Reward comparison under an identical training setup. Two policies are post-trained from the same pretrained RIDE with the same data, optimizer, and schedule, and differ only in the reward; each is shown at the last checkpoint of its run. The structural-only reward matches the scope of existing RNA design RL rewards and is defined in Appendix B . These results come from the controlled experiment’s own evaluation run, so some entries differ from Tables 1 – 3 . Bold marks the best of the three for metrics with a preferred direction; MFE, GC fraction, and within-target Hamming diversity are shown for context.

Figure 3: Structural-score trends over all 40 evaluations. Thick lines show a centered five-point moving average; faint traces show the raw held-out scores recorded during training, with one sampled candidate per target. Near the two ends, the average uses only the available points. Table 10 reports the evaluation of the last checkpoint, update 390. Appendix E.1 defines the smoothing.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Measurement	$a_{j}$	$b_{j}$	$w_{j}$
GDT-TS	0.20	0.75	0.25
TM-score	0.20	0.70	0.30
RMSD (Å)	12.0	2.0	0.30
Secondary-structure MCC	0.30	0.85	0.05
Ensemble defect / nucleotide	0.60	0.15	0.05
Composition deviation	0.50	0.02	0.05

Appendix

Table 5: Six-component reward parameters. Anchors

a_{j},b_{j}

identify low and high desirability. Equation ( 4 ) uses floor

10^{-3}

. All three structural measurements must be finite; available auxiliary terms determine the active weights.

Setting	Value
Base / parameters	Pretrained RIDE / 10,214,213
Training / evaluation targets	527 / 153
Epochs / updates	3 / 390
Targets / candidates per update	4 targets $\times$ 8 candidates = 32
Learning rate / minimum ratio	$2\times 10^{-4}$ / 0.3; 5% linear warmup, then cosine
Hardware / synchronization	Four GPUs / distributed gradient averaging

Appendix

Table 6: Settings of the reported run.

Model	$n$ (Precision)	$n$ (Recall)	$n$ (F1)	$n$ (MCC)
RiboDiffusion	133	137	137	133
gRNAde ( $T=0.1$ )	132	137	137	132
RDesign	135	137	137	135
RIDE	133	137	137	133
MORAD	136	137	137	136

Appendix

Table 7: Valid target counts for the candidate-0 pairing metrics. Counts correspond directly to Table 2 ; undefined measurements are excluded rather than replaced with zero. MORAD is update 390.

Model	Hamming diversity $\uparrow$	3-mer diversity $\uparrow$	Targets	Candidates
RiboDiffusion	0.2744	0.3964	153	1224
gRNAde ( $T=0.1$ )	0.1579	0.2406	153	1224
RDesign	0.4368	0.6184	153	1224
RIDE	0.2971	0.5270	153	1224
MORAD	0.2344	0.3673	153	1224

Appendix

Table 8: Sequence-diversity comparison. Each model uses 153 targets and 1,224 candidates (eight per target). Bold marks the highest reported diversity; higher diversity alone does not establish greater structural accuracy.

Metric	Ribo Diffusion	gRNAde $T=0.1$	gRNAde $T=0.8$	RDesign	RIDE	MORAD	Native
MFE $\sim$	-21.716	-28.092	-25.186	-28.863	-23.614	-30.149	-21.261
EFE $\sim$	-22.847	-28.918	-26.212	-29.708	-24.643	-30.937	—
ED $\downarrow$	43.653	26.913	28.486	25.502	29.184	21.403	28.049
NED $\downarrow$	0.6056	0.3458	0.3830	0.3349	0.3906	0.2830	0.3741
EDiv $\sim$	10.427	7.471	8.252	7.187	7.512	5.605	8.685
PE $\downarrow$	0.4284	0.2652	0.3260	0.2775	0.3059	0.2102	0.3581

Appendix

Table 9: Complete thermodynamic measurements on 153 targets. MFE/EFE: kcal/mol; ED: nt; NED: dimensionless; EDiv: mean base-pair distance; PE: positional entropy; MFEfreq: probability;

T_{m}

: ∘ C; GC: fraction. MORAD is update 390. Bold marks the best of the six models in the indicated direction; Native is a reference and is not ranked. Dashes denote unreported reference values;

\sim

marks metrics without a preferred direction, shown for context.

Figure 4: Complete RMSD trajectory. All 40 held-out measurements are connected in time order. Lower RMSD indicates closer agreement after alignment; the vertical marker shows update 390.

Checkpoint	GDT-TS $\uparrow$	TM(C4 $\prime$ ) $\uparrow$	RMSD (Å) $\downarrow$	C1 $\prime$ TM $\uparrow$	Recovery $\uparrow$
Step 180	0.3682 $\pm$ 0.0177	0.3218 $\pm$ 0.0198	8.263 $\pm$ 0.490	0.3716 $\pm$ 0.0171	0.5074 $\pm$ 0.0108
Step 230	0.3617 $\pm$ 0.0184	0.3208 $\pm$ 0.0201	8.733 $\pm$ 0.554	0.3697 $\pm$ 0.0176	0.5021 $\pm$ 0.0107
Step 340	0.3686 $\pm$ 0.0185	0.3271 $\pm$ 0.0205	8.435 $\pm$ 0.530	0.3771 $\pm$ 0.0177	0.5032 $\pm$ 0.0104
MORAD (390)	0.3774 $\pm$ 0.0179	0.3338 $\pm$ 0.0202	7.981 $\pm$ 0.489	0.3833 $\pm$ 0.0177	0.5102 $\pm$ 0.0106

Appendix

Table 10: Checkpoint comparison. Means over eight candidates per target, with standard errors, for four checkpoints from an earlier evaluation run; they differ from the candidate-0 results in Tables 1 and 2 , which also report update 390, the last checkpoint.

Explore similar work

Feb 18, 2026

RIDER: 3D RNA Inverse Design with Reinforcement Learning-Guided Diffusion

The inverse design of RNA three-dimensional (3D) structures is crucial for engineering functional RNAs in synthetic biology and therapeutics. While recent deep learning approaches have advanced this field, they are typically optimized and evaluated using native sequence recovery, which is a limited surrogate for structural fidelity, since different sequences can fold into similar 3D structures and high recovery does not necessarily indicate correct folding. To address this limitation, we propose RIDER, an RNA Inverse DEsign framework with Reinforcement learning that directly optimizes for 3D structural similarity. First, we develop and pre-train a GNN-based generative diffusion model conditioned on the target 3D structure, achieving a 9% improvement in native sequence recovery over state-of-the-art methods. Then, we fine-tune the model with an improved policy gradient algorithm using four task-specific reward functions based on 3D self-consistency metrics. Experimental results show that RIDER improves structural similarity by over 100% across all metrics and discovers designs that are distinct from native sequences.

Tianmeng Hu, Yongzheng Cui, Biao Luo +1

University of Exeter · Central South University

Inverse Design 3D Diffusion Models RL for Diffusion Models Conditional Diffusion Models

Ask PDF

Sep 29, 2026

RNA Design via Conditioned Flow Matching and Finite-Policy Reinforcement Learning

RNA design aims to identify sequences that fold into specified secondary structures. Existing methods formulate the task as target-specific search or conditional generation. However, natural RNA evolution proceeds through sequence variation and selection, with compensatory substitutions, whereas these methods do not explicitly model this process. To address this limitation, we propose a two-stage framework comprising RNA Inverse-Folding Flow (RNA-IFlow) and RNA-IFlow-RL. RNA-IFlow uses structure-conditioned Dirichlet Flow Matching to model coordinated variation across the sequence, while RNA-IFlow-RL maps the learned flow to a pairing-preserving finite policy and refines it with thermodynamic feedback. Our framework achieves leading performance on multiple benchmarks, reaching 85.19% Pass@1 on Rfam-27. Further analyses reveal thermodynamic gains, policy dynamics, and robustness across settings. Our work couples coordinated variation with thermodynamic selection, offering a novel paradigm for RNA design.

Zefeng Lin, Xianyong Fang, Tianfan Fu +1

School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Technology, Anhui University, Hefei, Anhui, China · State Key Laboratory for Novel Software Technology at Nanjing University, School of Computer Science, Nanjing University, Nanjing, Jiangsu, China

Inverse Design Molecular Generation Flow Matching RL for Generative Models

Ask PDF

Oct 8, 2026

La-Ribo: RNA Co-Design via Geometry-Latent Flow Matching

RNA function arises from the coupling of nucleotide sequence and three-dimensional structure, motivating their joint design. Coordinating global folding with nucleotide-level detail remains challenging under limited structural supervision. We introduce La-Ribo, a generative framework for RNA sequence-structure co-design via geometry-latent flow matching. La-Ribo retains a sparse phosphate-sugar--base scaffold and encodes nucleotide identity and local conformation in residue-wise latents. A shared flow network generates both jointly, and an RNA-specific decoder then reconstructs all heavy atoms. To expand supervision, we construct a quality-controlled corpus of 168,561 RNA structures, integrating experimental data with predictions from three folding models, including 10,631 MSA-supported structures generated in this work. La-Ribo improves designability and codesignability over the evaluated baselines across sampling budgets and two refolding models, and the same prior supports scaffold-conditioned inverse folding without additional training.

Runze Ma, Will Hua, Shuangjia Zheng

Shanghai Jiao Tong University · Independent Researcher

Inverse Design Flow-Based Generative Modeling

Ask PDF