Learning Latent Protein Languages for Autoregressive Generation

Mahdi Pourmirzaei; Farzaneh Esmaili; Amir Ziashahabi; Mohammadreza Pourmirzaei; Dong Xu

cs.LGOct 2, 2026

Learning Latent Protein Languages for Autoregressive Generation

Authors: Mahdi Pourmirzaei, Farzaneh Esmaili, Amir Ziashahabi, Mohammadreza Pourmirzaei, Dong Xu

Organizations: University of Missouri · University of Southern California · Independent Researcher

Abstract

Autoregressive transformers remain comparatively weak for protein sequence and structure generation. We study the role of target representation: amino acid tokens encode residue identities without explicit contextual semantics, while backbone coordinates require a discrete representation in our framework. We introduce two learned latent protein languages. Protein Latent Language (PLL) maps sequences to a 4,096-state contextual alphabet built on a frozen ESM-2 encoder, with one token per residue. Structure Latent Language (SLL) adapts GCP-VQVAE Lite with auxiliary sequence and confidence supervision while retaining decoding to backbone coordinates. We separately pretrain autoregressive transformer models on PLL and SLL tokens using next-token prediction, yielding PLLM and SLLM. Under matched downstream sequence training, PLLM has a fitted compute-scaling exponent of 0.038 versus 0.020 for the amino acid autoregressive model. In unconditional sequence generation, PLLM reduces the fraction of samples below a heuristic 1.5-bit residue-composition entropy threshold by 54% relative to the amino acid model across sampling temperatures. For sequence-to-structure prediction, replacing the original GCP-VQVAE Lite tokenizer with SLL reduces best validation perplexity by 34% under matched training. For long proteins, latent-token sampling is approximately 1,000 times faster than MSA-based AlphaFold2 in our measurements. In backbone generation, SLLM compares favorably with other generative models on diversity and novelty. We also observe early signs that using SLLM's internal token confidence for inference-time sampling can improve sequence-to-structure prediction quality beyond a single decoded sample. These results position learned latent protein languages as a promising substrate for autoregressive transformer scaling and inference-time sampling in protein generation.

Figures & tables

Figure 1: PLL construction. Stage 1 trains a vector-quantized variational autoencoder (VQ-VAE)-style latent reconstructor for frozen ESM-2 representations. Stage 2 finalizes PLL as a VQ-RAE by freezing the latent encoder and codebook and training a reinitialized transformer decoder with a linear residue-classification head.

Figure 2: SLL construction. A GCPNet and transformer encoder convert backbone geometry into discrete structure codes. The decoder reconstructs backbone coordinates, while auxiliary heads train the codes to retain residue identity, confidence, and sequence-representation information.

Figure 3: Sequence-to-structure generation via SLLM. E1 encodes the input sequence. The decoder predicts SLL tokens, which the SLL decoder maps to coordinates. Green, yellow, and blue denote protein embeddings, special, boundary, and condition tokens, and SLL tokens.

Configuration	Functional AUROC (%) $\uparrow$	Physicochemical Spearman (%) $\uparrow$
Base	58.26	23.93
MSE + distance	61.28	30.32
pLDDT pre-dec.	62.65	28.95
Inverse folding 0.1	67.15	23.66
Combined	72.37	32.30

Table 1: SLL supervised PST ablation averages.

Metric	FoldSeek [ 46 ]	ProTokens [ 24 ]	VanillaVQ [ 54 ]	AIDO [ 58 ]	ESM3 [ 16 ]	AminoAseed [ 54 ]	GCP-VQVAE Large [ 38 ]	GCP-VQVAE Lite [ 38 ]	GCP-VQVAE Lite 2
Functional AUROC (%) $\uparrow$	51.90	65.37	68.30	69.38	69.24	72.43	61.19 (68.10, 73.57)	61.84 (68.30, 74.93 )	71.32 (71.65, 78.08 )
Physicochemical Spearman (%) $\uparrow$	7.80	9.84	33.49	21.81	37.35	38.08	16.61 (17.89, 21.37)	20.84 (22.73, 23.89)	29.63 ( 29.71 , 27.52)

Table 2: Average supervised PST comparison with baseline tokenizers. Lite 2 denotes SLL, and parentheses give the four-layer MLP and transformer probe results for GCP-VQVAE variants [ 38 ] . Gray columns provide external context and are not used for bold and underline ranking.

Metric	GCP-VQVAE Lite 2	GCP-VQVAE Large [ 38 ]	GCP-VQVAE Lite [ 38 ]	AIDO [ 58 ]	ESM-3 VQVAE [ 16 ]	DPLM-2 VQVAE [ 49 ]	FoldToken 4 [ 12 ]	Gaujac et al. [ 14 ]	AminoAseed [ 54 ]
Average TM-score $\uparrow$	0.9456	0.9844	0.9687	0.9470	0.9291	0.8311	0.8788	0.8394	0.4000
Average RMSD $\downarrow$	1.4722	0.6172	0.9364	1.8189	2.8905	7.3650	3.0106	6.0002	18.2178

Table 3: Average reconstruction quality over CASP14, CASP15, CASP16, CAMEO-2024, and the zero-shot set from the original benchmark [ 38 ] , where Lite 2 is our SLL tokenizer. The grayed GCP-VQVAE Large column is not used for bold and underline ranking.

Figure 4: Matched sequence-to-structure tokenizer comparison. Left: train loss. Right: validation perplexity, where Lite 2 lowers the best value by 34%. The learning-speed annotations compare training progress by epoch.

Panel of Figure 5 — Figure 5: Validation loss versus estimated downstream autoregressive training compute. PLLM has a steeper fitted slope than the amino acid model under matched sequence training. SLLM is shown separately with structure-token data and different training exposure.

Panel of Figure 6 — Figure 6: Amino acid versus PLLM sequence-generation diagnostics. Left: mean decoded residue-composition entropy across unconditional sampling temperatures. Middle: unconditional length distribution at $T=1.0$ . Right: length-conditioned entropy-pLDDT correlation after entropy-filtered resampling.

Table 10

Panel of Figure 7 — Figure 7: Sequence-to-structure inference-time scaling on the held-out CAMEO-2024 set as a function of the number of sampled SLL trajectories per input sequence ( $k$ ). Oracle best@ $k$ is a ceiling that picks the best candidate after coordinate decoding, while DeepConf consensus@ $k$ uses only internal SLL-token confidence and token-space consensus to pick a single candidate before any coordinate decoding. (a) lDDT-CA $\uparrow$ . (b) RMSD $\downarrow$ . Both metrics improve overall as the sampling budget increases, and DeepConf recovers a meaningful fraction of the oracle gain. More details are given in Section C.3.4 .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Stage 1	Stage 2
Objective	$\mathcal{L}_{\mathrm{mse}}$ , $\mathcal{L}_{\mathrm{cos}}$ , $\mathcal{L}_{\mathrm{commit}}$	$\mathcal{L}_{\mathrm{class}}$
Epochs	$64$	$8$
Maximum sequence length	$1\,024$	$1\,024$
Training size	$2.0$ M	$20.0$ M
Batch size	$512$ ( $16$ per GPU)	$2\,048$ ( $64$ per GPU)
Optimizer	AdamW, weight decay 0.01, gradient clipping $1.0$	Same as Stage 1

Appendix

Table 6: PLL training choices for the two tokenizer stages. Stage 1 learns the latent bottleneck, and Stage 2 freezes the latent encoder and codebook and trains the transformer decoder for residue recovery. Maximum sequence length includes the beginning-of-sequence and end-of-sequence tokens from the ESM tokenizer.

Setting	Value
Active objective terms	$\mathcal{L}_{\mathrm{coord}}$ , $\mathcal{L}_{\mathrm{dist}}$ , $\mathcal{L}_{\mathrm{vq}}$ , $\mathcal{L}_{\mathrm{if}}$ , $\mathcal{L}_{\mathrm{pLDDT}}$ , $\mathcal{L}_{\mathrm{ESM}}$
Epochs	$64$
Maximum sequence length	$1\,280$ residues
Training size	$2.28$ M
Batch size	$512$ ( $16$ per GPU)
Precision	bfloat16

Appendix

Table 7: SLL tokenizer training choices for the final GCP-VQVAE Lite 2 model. The architecture and source dataset follow GCP-VQVAE Lite.

Type	Tokens
Special	<PAD> , <BOS> , <EOS> , <MASK> , <UNK>
Amino acid	A C D E F G H I K L M N P Q R S T V W Y X
PLL	learned code IDs $0$ to $4\,095$
SLL	learned structure code IDs $0$ to $4\,095$
Length condition	<LEN_n>
Protein boundary	<BOP> , <EOP>

Appendix

Table 8: Token inventory used by the autoregressive tokenizer.

Case	Stream format
Amino acid	<BOS> A C D $\ldots$ Y <EOS>
PLL	<BOS> $P_{1}$ $P_{2}$ $\ldots$ $P_{L}$ <EOS>
SLL	<BOS> $S_{1}$ $S_{2}$ $\ldots$ $S_{L}$ <EOS>
Amino acid length condition	<BOS> <LEN_n> A C D $\ldots$ Y <EOS>
PLL length condition	<BOS> <LEN_n> <BOP> $P_{1}$ $\ldots$ $P_{L}$ <EOP> <EOS>
SLL length condition	<BOS> <LEN_n> <BO3D> $S_{1}$ $\ldots$ $S_{L}$ <EO3D> <EOS>

Appendix

Table 9: Example token streams for the autoregressive experiments.

P_{i}

denotes a PLL code token, and

S_{i}

denotes an SLL code token.

Size label	Width	Layers	Heads	KV heads	Head dim	FFN dim
17.9M	$512$	$6$	$4$	$4$	$64$	$2\,048$
51.8M	$640$	$12$	$5$	$5$	$64$	$2\,560$
109.7M	$768$	$19$	$6$	$3$	$64$	$3\,072$
263.4M	$1\,024$	$26$	$8$	$4$	$64$	$4\,096$
611.8M	$1\,536$	$27$	$12$	$6$	$64$	$6\,144$
1.33B	$2\,048$	$34$	$16$	$4$	$64$	$8\,192$

Appendix

Table 10: Decoder-only transformer grid used for the scaling-law runs. The head dimension is

64

throughout.

Setting	Amino acid model and PLLM	SLLM
Training data	UniRef50	AFDB $\cap$ UniRef50 [ 38 ]
Epochs	$4$	$8$
Steps per epoch	$128$ k	$128$ k
Total tokens seen per model	$70.7$ B	$81.5$ B
Batch size	$512$ ( $16$ per GPU)	$512$ ( $16$ per GPU)
Precision	bfloat16	bfloat16

Appendix

Table 11: Training hyperparameters for the scaling-law runs.

Length	Model	pLDDT $\uparrow$	Pair. TM $\downarrow$	Clust. $\uparrow$	Max TM $\downarrow$
100	Native ref	68.46	0.5500	0.7500	–
100	ProGen2 (700M) ref	64.00	0.4200	0.9400	0.6400
100	DPLM (650M) ref	85.38	0.5000	0.8000	0.7400
100	ESM3 (1.4B) ref	54.26	0.4500	0.9000	0.6800
100	PLLM (1.3B)	42.74	0.2825	0.9800	0.4189
100	PLM (1.3B)	64.75	0.3019	0.8400	0.4741

Appendix

Table 12: Length-wise sequence ProteinBench. Rows marked ref are taken from ProteinBench [ 52 ] . Our PLM and PLLM use

T=0.5

, retain

50

sequences with

H_{\mathrm{seq}}(x)>1.5

bits per length, and use ESMFold.

Length	Model	scTM $\uparrow$	scRMSD $\downarrow$	TM similarity $\downarrow$	Clust. $\uparrow$	Max TM $\downarrow$
50	Native ∗	0.91	0.74	0.29	0.66	N/A
50	Genie ∗ [ 25 ]	0.57	3.12	0.32	0.90	0.57
50	RFdiffusion ∗ [ 50 ]	0.95	0.45	0.58	0.67	0.65
50	Protpardelle [ 32 ]	0.9655	0.5785	0.7369	0.3200	0.7608
50	ProteinGenerator [ 27 ]	0.7984	1.2202	0.8938	0.1300	0.3468
50	PLAID [ 30 ]	0.8028	1.3188	0.7012	0.5600	0.6463

Appendix

Table 13: Structure-design results on ProteinBench [ 52 ] at target lengths 50 and 100. ∗ Results taken from ProteinBench. † DPLM-2 jointly models amino acid sequences and structure tokens.

Length	Model	scTM $\uparrow$	scRMSD $\downarrow$	TM similarity $\downarrow$	Clust. $\uparrow$	Max TM $\downarrow$
300	Native ∗	0.97	0.82	0.28	0.77	N/A
300	Genie ∗ [ 25 ]	0.27	20.37	0.23	1.00	0.30
300	RFdiffusion ∗ [ 50 ]	0.96	1.03	0.36	0.65	0.64
300	Protpardelle [ 32 ]	0.9603	1.3968	0.7395	0.1500	0.8152
300	ProteinGenerator [ 27 ]	0.8926	2.0403	0.6409	0.2600	0.7158
300	PLAID [ 30 ]	0.7474	3.0922	0.5950	0.5700	0.7085

Appendix

Table 14: Structure-design results on ProteinBench [ 52 ] at target lengths 300 and 500. ∗ Results taken from ProteinBench. † DPLM-2 jointly models amino acid sequences and structure tokens.

Data source	Role	Approx. samples
UniRef50 [ 43 ]	PLL tokenizer; PLLM and amino acid generation	$65.78$ M
AFDB representatives	SLL tokenizer	$2.28$ M
AFDB $\cap$ UniRef50 following GCP-VQVAE [ 38 ]	SLLM generation	$38.15$ M

Appendix

Table 15: Main data sources.

Figure 8: UniRef50 reference statistics after truncating sequences to 1022 residues. These distributions provide the reference length and amino acid composition statistics used when analyzing de novo amino acid and PLLM generation.

Residue	Accuracy (%)	Residue	Accuracy (%)	Residue	Accuracy (%)
A	$99.89$	I	$99.81$	R	$99.81$
C	$99.18$	K	$99.77$	S	$99.87$
D	$99.83$	L	$99.91$	T	$99.86$
E	$99.87$	M	$99.48$	V	$99.87$
F	$99.72$	N	$99.79$	W	$99.20$
G	$99.86$	P	$99.84$	Y	$99.69$

Appendix

Table 16: Stage 2 PLL per-amino-acid recovery on the UniRef50 validation split.

	Functional AUROC (%) $\uparrow$									Physicochemical Spearman (%) $\uparrow$
Configuration	BindInt	BindBio	BindShake	CatInt	CatBio	Con	Rep	Ept	Avg	RMSF	BFact	NEQ	Avg
Base	56.48	59.36	71.84	60.02	54.90	58.21	54.97	57.10	58.26	34.66	22.89	14.26	23.93
ESM pre-decoder	57.91	61.15	70.00	61.91	60.18	63.63	59.07	54.70	60.47	35.55	21.90	16.84	24.76
ESM post-decoder	58.94	61.86	73.03	63.11	58.12	61.18	57.07	57.88	60.62	36.70	24.20	14.88	25.26
Inverse folding 0.01	58.00	55.85	72.42	59.29	54.18	58.63	54.85	59.81	58.24	34.79	23.59	16.63	25.00
Inverse folding 1.0	67.18	88.90	63.38	63.06	86.18	62.00	55.07	58.48	68.34	25.96	17.15	21.49	21.53

Appendix

Table 17: Supervised PST ablation results for SLL. Task columns average available splits, and Avg averages all rows within each category. Cells are colored relative to the Base row. Gray rows mark runs that did not converge cleanly.

Task	Split	FoldSeek [ 46 ]	ProTokens [ 24 ]	VanillaVQ [ 54 ]	AIDO [ 58 ]	ESM3 [ 16 ]	AminoAseed [ 54 ]	GCP-VQVAE Large [ 38 ]	GCP-VQVAE Lite [ 38 ]	GCP-VQVAE Lite 2
Functional Site Prediction, AUROC (%) $\uparrow$
BindInt	Fold	53.18	44.66	47.25	44.66	44.30	47.11	52.61 (50.26, 52.85)	49.43 (49.00, 55.03 )	52.60 ( 53.73 , 50.85)
BindInt	SupFam	46.20	86.05	86.71	84.21	90.77	90.53	58.04 (70.61, 80.67)	62.27 (70.27, 80.77)	79.59 (79.84, 93.86 )
BindBio	Fold	52.37	58.47	62.02	65.50	62.84	65.73	75.05 (92.74, 84.01)	80.93 ( 94.61 , 91.74)	93.97 ( 94.28 , 87.99)
BindBio	SupFam	52.41	60.47	62.92	66.70	65.22	68.30	75.64 (93.40, 86.62)	81.72 ( 94.08 , 90.08)	93.95 ( 94.10 , 88.52)
BindShake	Org	53.40	59.82	67.04	69.28	66.10	69.61	66.97 (67.96, 72.71 )	66.34 (67.11, 68.08)	70.81 (70.79, 75.75 )

Appendix

Table 18: Supervised PST full-run comparison between SLL and structure-tokenizer baselines. GCP-VQVAE Lite 2 denotes SLL, and parentheses report four-layer MLP and transformer probe results for GCP-VQVAE variants.

Dataset	Metric	GCP-VQVAE Lite 2	GCP-VQVAE Large [ 38 ]	GCP-VQVAE Lite [ 38 ]	AIDO [ 58 ]	ESM-3 VQVAE [ 16 ]	DPLM-2 VQVAE [ 49 ]	FoldToken 4 [ 12 ]	Gaujac et al. [ 14 ]	AminoAseed [ 54 ]
CASP14	TM-score $\uparrow$	0.9569	0.9890	0.9751	0.9831	0.9734	0.9056	0.8989	0.8940	0.3951
CASP14	RMSD $\downarrow$	1.1343	0.5431	0.8435	0.6587	1.0968	2.0896	2.0027	1.8913	14.6739
CASP15	TM-score $\uparrow$	0.9390	0.9884	0.9665	0.9660	0.9418	0.8268	0.9136	0.7911	0.3831
CASP15	RMSD $\downarrow$	2.1261	0.5293	0.9219	0.8611	1.3407	6.3826	1.6856	8.5592	18.5827
CASP16	TM-score $\uparrow$	0.9542	0.9857	0.9757	0.8805	0.9103	0.7852	0.8211	0.8345	0.3433
CASP16	RMSD $\downarrow$	1.5030	0.7567	1.0614	4.9773	4.4918	14.1934	5.4115	6.9375	25.9508

Appendix

Table 19: Full reconstruction comparison following GCP-VQVAE [ 38 ] . GCP-VQVAE Lite 2 denotes SLL. The grayed GCP-VQVAE Large column is not used for bold and underline ranking.

Model	ApoHolo		FoldSwitching
Model	PCC (%) $\uparrow$	Spearman (%) $\uparrow$	PCC (%) $\uparrow$	Spearman (%) $\uparrow$
ProteinMPNN [ 8 ]	35.80	45.22	51.46	55.91
MIF [ 51 ]	35.82	43.55	54.48	59.27
FoldSeek [ 46 ]	34.79	43.03	51.88	56.54
ProTokens [ 24 ]	43.32	54.05	61.20	65.90
ESM3 [ 16 ]	39.76	50.97	57.12	62.23

Appendix

Table 20: Unsupervised PST full-run comparison following GCP-VQVAE [ 38 ] . GCP-VQVAE Lite 2 denotes SLL. The grayed GCP-VQVAE Large row is not used for bold and underline ranking.

Figure 9: Unconditional sequence-entropy distributions for PLLM and amino acid generation across sampling temperatures. PLLM approaches the UniRef50-like high-entropy regime around

T=0.6

T=1.0

Figure 10: Unconditional length distributions for PLLM and amino acid generation across sampling temperatures. Direct amino acid generation overproduces capped sequences, while PLLM is closest to the UniRef50 length regime around

T=0.6

T=1.0

Figure 11: Length-conditioned mean sequence entropy across temperatures and target lengths. The green line marks the UniRef50 mean entropy, and the red line marks the drift threshold.

Figure 12: Length-conditioned sequence-length distributions across target lengths and temperatures. Both modalities concentrate near the requested length, showing that the length prefix largely fixes the stopping and target-size issue.

Figure 13: Length-conditioned sequence-entropy distributions across target lengths and temperatures. PLL shifts into the high-entropy natural-protein regime more consistently than amino acid generation.

Figure 14: Length-conditioned entropy-pLDDT correlation without removing drifted sequences.

Figure 15: SLL length-conditioned continuation after unconditional pretraining. A two-epoch curriculum with length tokens on 50% of samples lowers unconditional validation loss and adds a length-control interface for generation.

Panel of Figure 16 — Figure 16: Sequence-to-structure inference-time scaling on the combined CASP14, CASP15, and CASP16 tuning set. DeepConf uses internal confidence without ground truth, while oracle best@k is a ceiling on the sampled candidate pool.

Panel of Figure 17 — Figure 17: Sequence-to-structure inference-time scaling on CAMEO-2024 after fixing the selector on CASP14/15/16.

Explore similar work

AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design
Jun 1, 2026Sahil Rahman, Maxx Richard RahmanRL for Language Models Direct Preference Optimization
Bedford College, London, United Kingdom · Saarland University, Saarbrücken, Germany
Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Feb 24, 2026Anna Hart, Chi Han, Jeonghwan Kim +2Protein Language Models Protein Representation Learning
Siebel School of Data Science and Computing, University of Illinois Urbana-Champaign, 201 N Goodwin Ave, Urbana, IL 61801, USA · DOE Center for Advanced Bioenergy and Bioproducts Innovation, University of Illinois Urbana-Champaign, 1206 W Gregory Drive, Urbana, IL 61801, USA · Chemical and Biomolecular Engineering, University of Illinois Urbana-Champaign, 600 S Matthews Ave, Urbana, IL 61801, USA
AMix-2: Establishing Protein as a Native Modality in Large Language Models
May 29, 2026Keyue Qiu, Yixin Wu, Lihao Wang +19Multimodal Large Language Models Protein Language Models
1Shanghai Artificial Intelligence Laboratory · Institute for AI Industry Research (AIR), Tsinghua University

Jun 1, 2026J/K move · Enter open · S save

AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design

Sahil Rahman, Maxx Richard Rahman

Bedford College, London, United Kingdom · Saarland University, Saarbrücken, Germany

RL for Language Models Direct Preference Optimization Protein Language Models LLM Tool Use Molecular Generation Protein Design

Open paper AskPDF arXiv

Protein language models (PLMs) are passive oracles: they generate sequences in a single forward pass with no mechanism to consult external biophysical feedback or redirect generation when a candidate violates thermodynamic or structural constraints. We introduce AgentPLM, which addresses this by equipping a pre-trained PLM with i) Reasoning-Augmented Decoding (RAD), which interleaves autoregressive generation with tool calls (ESMFold, FoldX, AutoDock Vina), and ii) Contrastive Agent Policy Optimisation (CAPO), a trajectory-level extension of direct preference optimisation that trains the policy end-to-end to learn when oracle feedback is informative rather than merely imitating high-fitness sequences. We evaluate AgentPLM on benchmark tasks spanning de novo enzyme design, antibody optimisation, thermostability, PPI interface design, and zero-shot fitness prediction with standardised oracle APIs and controlled sequence-identity splits. AgentPLM achieves state-of-the-art results with a gain in antibody top-10% hit rate over the strongest passive baseline, providing mechanistic evidence of online error correction without explicit backtracking.

Abstract

Explore similar work

AgentPLM: Agentic Protein Language Models with Reasoning-Augmented Decoding for Protein Sequence Design

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference

AMix-2: Establishing Protein as a Native Modality in Large Language Models