Unlocking Multimodal Protein Language Models at Inference Time

Yi Zhou; Qipeng Wang; Yunqing Liu; Jun Xia; Qing Li; Wenqi Fan

cs.CEAug 26, 2026

Unlocking Multimodal Protein Language Models at Inference Time

Authors: Yi Zhou, Qipeng Wang, Yunqing Liu, Jun Xia, Qing Li, Wenqi Fan

Organizations: The Hong Kong Polytechnic University · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Multimodal protein language models (pLMs) learn joint protein sequence-structure distributions, and their generation performance should also depend critically on inference-time sampling strategies. Yet prior work has focused more on model training than on how inference-time strategies behave. In this paper, we establish a three-stage investigation framework to empirically study the inference design space of multimodal pLMs across three representative pLMs and four fundamental tasks. We evaluate vanilla sampling, task-specific classifier-free guidance, and reward-guided beam search on multimodal pLMs, corresponding to controls over sampling distributions, per-step logits, and parallel trajectories. Throughout the complementary advancements centered on exploration-exploitation trade-off, we (1) reveal the suboptimality of default inference protocols and identify task-oriented sampling preferences; (2) observe substantial quantitative gains across tasks, consistently boosting the upper bound performance of multimodal pLMs without updating model parameters; (3) derive conclusions about base models that differ from prior consensus.

Figures & tables

Figure 1: A structure prediction case. A . Ground-truth. B . One-step argmax prediction result. C . $L$ -step temperature-based sampling result.

Figure 2: Overview of our empirical investigation framework. A. Multimodal protein language models (pLMs) perform four foundational protein modeling tasks via iterative diffusion sampling. B. At the distribution level, we benchmark vanilla sampling across three pLMs and four tasks, therefore identifying the task-oriented exploration-exploitation trade-off at inference time. C. At the logit level, we propose task-motivated classifier-free guidance that steers each step’s logits and therefore directs the sampling trajectory. D. At the trajectory level, reward-guided beam search navigates parallel trajectories using model-internal quality signals, highlighting the effectiveness of “explore-then-select” inference across tasks.

Figure 3: Vanilla sampling benchmark . Panels A - D show ESM3 performance on each task under the optimal vanilla sampling configuration, with ablations over total steps, per-step randomness (unmasking $\times$ temperature), temperature annealing, remasking, and multimodal sampling order. The optimal configuration is highlighted, and the default performance is marked for reference.

		Designable Subset		All Samples
Model	Cost*	#Design $\uparrow$	#Clusters $\uparrow$	pLDDT $\uparrow$	scTM $\uparrow$	#Clusters $\uparrow$	$\alpha$ / $\beta$ %
DPLM-2 ${}_{\text{Default}}$	2.18, 1.17	149.0 (126, 160)	76.0 (62, 84)	81.920 ± 8.643	0.906 ± 0.105	262.0 (248, 273)	38.4 / 16.9
DPLM-2 ${}_{\text{CFG}}$	6.54, 3.22	162.4 (157, 168)	78.4 (75, 84)	82.242 ± 8.247	0.913 ± 0.091	247.4 (241, 254)	36.9 / 17.4
DPLM-2.1 ${}_{\text{Default}}$	2.15, 1.02	143.4 (129, 153)	86.6 (82, 97)	85.116 ± 7.681	0.904 ± 0.106	304.2 (295, 312)	43.6 / 13.6
DPLM-2.1 ${}_{\text{CFG}}$	6.44, 3.00	162.2 (145, 171)	96.4 (90, 104)	84.995 ± 7.671	0.911 ± 0.099	284.8 (272, 293)	42.5 / 14.2
ESM3 ${}_{\text{Default seq2struct}}$	1.25, 1.68	52.8 (31, 70)	34.4 (24, 49)	61.387 ± 17.563	0.660 ± 0.242	381.4 (371, 396)	48.3 / 4.3

Table 1: Evaluation of Unconditional Sequence-Structure Co-Generation. Note: Improves over the last investigation stage.

Figure 4: Qualitative Analysis. A-B. Optimal vanilla sampling and CFG alleviate the abnormal properties of ESM3-generated proteins. C. Visualizations of representative cases.

Model	Cost*	#Solved $\uparrow$	Succ. Rate $\uparrow$	#Clusters $\uparrow$
DPLM-2 ${}_{\text{Default}}$	2.88, 1.37	15.6 (15, 16)	15.3 ± 0.3 %	72.6 (63, 83)
DPLM-2 ${}_{\text{Vanilla}}$	0.47, 0.42	16.8 (16, 17)	20.6 ± 0.5 %	72.8 (68, 80)
DPLM-2 ${}_{\text{CFG}}$	1.41, 0.75	15.8 (15, 17)	20.4 ± 0.2%	71.6 (63, 81)
DPLM-2.1 ${}_{\text{Default}}$	2.83, 1.15	14.2 (14, 15)	18.7 ± 0.6 %	51.0 (45, 55)
DPLM-2.1 ${}_{\text{Vanilla}}$	2.83, 1.15	15.4 (15, 16)	19.4 ± 0.6 %	66.8 (61, 72)
DPLM-2.1 ${}_{\text{CFG}}$	8.49, 3.33	15.0 (14, 16)	19.3 ± 0.5 %	61.2 (56, 68)

Table 2: Evaluation of Motif-Scaffolding.

CAMEO

PDB Date Split

CAMEO

PDB Date Split

Model

Cost*

RMSD

\downarrow

TM-score

\uparrow

RMSD

\downarrow

TM-score

\uparrow

Model

Cost*

scTM

\uparrow

AAR

scTM

\uparrow

AAR

DPLM-2

{}_{\text{Default}}

18.5, 6.2

7.483±6.126

0.786±0.170

5.253±5.143

0.836±0.144

DPLM-2

{}_{\text{Default}}

18.5, 6.2

0.870±0.158

48.1

0.912±0.111

55.6

DPLM-2

{}_{\text{Vanilla}}

5.6, 2.8

7.513±6.178

0.790±0.167

5.249±5.103

0.837±0.141

DPLM-2

{}_{\text{Vanilla}}

0.2, 0.1

0.875±0.153

48.2

0.917±0.111

55.6

DPLM-2

{}_{\text{CFG}}

11.2, 4.1

7.414±6.112

0.786±0.169

5.178±5.029

0.835±0.142

DPLM-2

{}_{\text{CFG}}

0.4, 0.1

0.877±0.151

47.1

0.922±0.101

54.9

DPLM-2.1

{}_{\text{Default}}

18.2, 5.5

6.297±6.181

0.822±0.170

DPLM-2.1

{}_{\text{Default}}

18.2, 5.5

0.875±0.144

53.3

DPLM-2.1

{}_{\text{Vanilla}}

1.5, 0.5

6.259±6.051

0.824±0.167

DPLM-2.1

{}_{\text{Vanilla}}

1.5, 0.5

0.879±0.142

53.4

Table 3: Evaluation of Protein Structure Prediction (Left) and Inverse Folding (Right).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	DPLM-2	DPLM-2.1	ESM3	ESM3
Multimodal Order	Synchronous	Synchronous	Seq $\rightarrow$ Struct	SS $\rightarrow$ Struct $\rightarrow$ Seq
Sampling Step $T$	500	500	$L$ ; $1$	$L$
Unmasking	Stochastic	Stochastic	Random	Random
Remasking	Enabled	Enabled	Disabled	Disabled
Temp. & Annealing	From 2.0 to 0.1	From 1.1 to 0.1	From 1.0 to 0.0; 0.0	0.7

Appendix

Table 4: Default Sampling Strategies for Unconditional Co-Generation

	DPLM-2	DPLM-2.1	ESM3
Multimodal Order	Synchronous	Synchronous	Seq $\rightarrow$ Struct
Sampling Step $T$	500	500	$L/2$ ; $L/8$
Unmasking	Stochastic	Stochastic	Random
Remasking	Enabled	Enabled	Disabled
Temp. & Annealing	From 2.0 to 1.0	From 1.1 to 0.1	From 1.0 to 0.0

Appendix

Table 5: Default Sampling for Motif-Scaffolding

	DPLM-2	DPLM-2.1	ESM3
Sampling Step $T$	100	100	1
Unmasking	Deterministic	Deterministic	Deterministic
Remasking	Enabled	Enabled	Disabled
Temp. & Annealing	0.0	0.0	0.0

Appendix

Table 6: Default Sampling for Structure Prediction

	DPLM-2	DPLM-2.1	ESM3
Sampling Steps $T$	100	100	8
Unmasking	Deterministic	Deterministic	Random
Remasking	Enabled	Enabled	Disabled
Temp. & Annealing	0.0	0.0	From 1.0 to 0.0

Appendix

Table 7: Default Sampling for Inverse Folding

	DPLM-2	DPLM-2.1	ESM3
Multimodal Order	Synchronous	Synchronous	Synchronous
Sampling Step $T$	500	500	$L$
Unmasking	Stochastic	Stochastic	Stochastic
Remasking	Enabled	Enabled	Enabled
Temp. & Annealing	From 2.0 to 0.1	From 1.1 to 0.1	From 1.0 to 0.0

Appendix

Table 8: Optimal Vanilla Sampling for Uncond. Co-Gen.

	DPLM-2	DPLM-2.1	ESM3
Multimodal Order	Synchronous	Synchronous	Synchronous
Sampling Step $T$	$L$	500	$L$
Unmasking	Stochastic	Stochastic	Random
Remasking	Enabled	Enabled	Enabled
Temp. & Annealing	From 0.5 to 0.0	From 0.5 to 0.0	From 0.7 to 0.0

Appendix

Table 9: Optimal Vanilla Sampling for Motif-Scaffolding

	DPLM-2	DPLM-2.1	ESM3
Sampling Step $T$	$L/8$	8	8
Unmasking	Random	Deterministic	Stochastic
Remasking	Enabled	Enabled	Enabled
Temp. & Annealing	From 0.1 to 0.0	From 0.1 to 0.0	From 0.5 to 0.0

Appendix

Table 10: Optimal Vanilla Sampling for Structure Prediction

	DPLM-2	DPLM-2.1	ESM3
Sampling Step $T$	1	8	8
Unmasking	Random	Stochastic	Deterministic
Remasking	Enabled	Enabled	Enabled
Temp. & Annealing	From 0.1 to 0.0	From 0.1 to 0.0	From 0.5 to 0.0

Appendix

Table 11: Optimal Vanilla Sampling for Inverse Folding

Figure 5: Vanilla sampling: optimal configurations and ablation results . A. Pareto comparison between designability and diversity in unconditional protein sequence-structure co-generation. The first legend shows the optimal configuration, and each subsequent legend changes one setting at a time. B-D. The green bars to the left of the vertical dotted line show ablations over the total sampling step $T$ , with the other vanilla sampling settings listed. The remaining bars show one-at-a-time changes to the optimal configuration: "no rand" denotes deterministic unmasking with temperature $=0.0$ (argmax sampling); "const temp" and "no remasking" denote disabled temperature annealing and remasking, respectively.

	Uncon. Co-Gen.	Motif-Scaffolding	Structure Prediction	Inverse Folding
DPLM-2	$w_{s}=2.0$ , $w_{z}=1.5$	$w_{s}=1.5$ , $w_{z}=1.0$	$w_{z}=1.5$	$w_{s}=4.0$
DPLM-2.1	$w_{s}=2.0$ , $w_{z}=1.5$	$w_{s}=1.5$ , $w_{z}=1.0$	$w_{z}=1.5$	$w_{s}=2.0$
ESM3	$w_{s}=2.0$ , $w_{z}=3.0$	$w_{s}=2.0$ , $w_{z}=2.0$	$w_{z}=2.0$	$w_{s}=2.0$

Appendix

Table 12: Selected CFG scales.

Figure 6: CFG scale analysis for protein structure prediction and inverse folding on CAMEO2022.

	#Solved / 24	Success Rate	#Clusters
ESM3 ${}_{\text{Vanilla}}$	21.6 (21, 22)	30.9 ± 1.1 %	110.0 (100, 124)
ESM3 ${}_{\text{CFG (Motif-only)}}$	22.6 (22, 23)	35.3 ± 0.6 %	146.8 (127, 158)
ESM3 ${}_{\text{CFG (Cross-Modal)}}$	22.4 (21, 23)	36.6 ± 1.4 %	155.0 (144, 166)
ESM3 ${}_{\text{CFG (Both)}}$	22.8 (22, 23)	37.5 ± 1.0 %	179.4 (165, 198)

Appendix

Table 13: Ablation study on motif-scaffolding CFG.

Figure 7: Two cases to illustrate ESM3’s RMSD-TM-score divergence in protein structure prediction.

		Designable Subset		All Samples
$N$	$B$	#Design	#Clusters	pLDDT	scTM	#Clusters	$\alpha$ / $\beta$ %
2	1	318.0	137.0	89.529	0.955	194.0	45.7 / 12.8
4	1	350.4	139.0	90.704	0.964	177.6	47.9 / 11.6
8	1	342.0	132.0	90.632	0.963	175.0	47.6 / 11.7
4	2	385.0	119.0	91.342	0.973	138.0	50.2 / 10.5
4	4	361.0	117.0	91.574	0.970	135.0	50.0 / 10.9

Appendix

Table 14: ESM3

{}_{\text{Beam}}

hyperparameter selection: unconditional co-generation

$N$	$B$	#Solved	Success Rate	#Clusters
1	2	23.0	38.2 %	174.0
1	4	23.0	38.0 %	186.0
2	1	23.0	43.7 %	222.0
2	2	23.0	46.0 %	240.0
2	4	23.0	47.8 %	233.0
4	1	23.0	49.9 %	261.0

Appendix

Table 15: ESM3

{}_{\text{Beam}}

hyperparameter selection: motif scaffolding

Figure 8: ESM3 ${}_{\text{Beam}}$ hyperparameter selection. Left: protein structure prediction; right: inverse folding.

Table 16: ESM3

{}_{\text{Beam}}

explore-then-select: unconditional protein sequence-structure co-generation

Temperature	#Solved / 24	Success Rate	#Clusters
From 0.7 to 0.0	23.0 (23, 23)	48.9 ± 0.9 %	214.6 (202, 226)
From 1.0 to 0.0	23.0 (23, 23)	49.9 ± 0.7 %	261.0 (250, 278)

Appendix

Table 17: ESM3

{}_{\text{Beam}}

explore-then-select: motif-scaffolding

	CAMEO		PDB Date Split
Umasking	scTM $\uparrow$	AAR	scTM $\uparrow$	AAR
Deterministic	0.910 ± 0.134	45.7	0.954 ± 0.075	49.1
Stochastic	0.914 ± 0.129	46.0	0.956 ± 0.069	49.3

Appendix

Table 18: ESM3

{}_{\text{Beam}}

explore-then-select: inverse folding

Explore similar work

Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference
Mar 9, 2026Noah Golowich, Fan Chen, Dhruv Rohatgi +4LLM Inference Inference-Time Guidance
Microsoft Research · MIT · NYU
How to make the most of your masked language model for protein engineering
Mar 11, 2026Calvin McCarter, Nick Bhattacharya, Sebastian W. Ober +1Protein Language Models Inference-Time Search
BigHat Biosciences
Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language Models Protein Representation Learning
Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA

Mar 9, 2026J/K move · Enter open · S save

Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference

Noah Golowich, Fan Chen, Dhruv Rohatgi, Raghav Singhal+3

Microsoft Research · MIT · NYU

LLM Inference Inference-Time Guidance Efficient Language Model Reasoning

Open paper AskPDF arXiv

Inference-time methods that aggregate and prune multiple samples have emerged as a powerful paradigm for steering large language models, yet we lack any principled understanding of their accuracy-cost tradeoffs. In this paper, we introduce a route to rigorously study such approaches using the lens of particle filtering algorithms such as Sequential Monte Carlo (SMC). Given a base language model and a process reward model estimating expected terminal rewards, we ask: how accurately can we sample from a target distribution given some number of process reward evaluations? Theoretically, we identify (1) simple criteria enabling non-asymptotic guarantees for SMC; (2) algorithmic improvements to SMC; and (3) a fundamental limit faced by all particle filtering methods. Empirically, we demonstrate that our theoretical criteria effectively govern the sampling error of SMC, though not necessarily its final accuracy, suggesting that theoretical perspectives beyond sampling may be necessary.

Abstract

Explore similar work

Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference

How to make the most of your masked language model for protein engineering

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference