cs.LGNov 10, 2026

Pyrite: Building and Benchmarking Protein-Ligand Pose Ranking Agents

Authors: Megha Srivastava, Michael H. Lim, Allison M. Keys, Pranav Murugan, Nate Gruver

Organizations: Stanford University · Genesis Molecular AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Predicting how a ligand binds to its target is central to small-molecule drug discovery. Modern cofolding models can generate diverse candidate poses, but reliably ranking these candidates remains challenging. We introduce Pyrite, a pairwise pose-ranking benchmark comprising 1,500 pose pairs across diverse protein-ligand complexes. Expert chemists equipped with 3D visualization tools and physics-based scores perform near chance, highlighting the difficulty of interpreting available chemical signals. To investigate automated ranking, we introduce PyriteRep, a structured representation of molecular geometry, protein-ligand interactions, and physicochemical scores, and develop two LLM-based agents: PyriteReasoner, which directly compares poses through text-based reasoning, and PyriteFeaturizer, which generates executable code to construct features for logistic regression, optionally incorporating features derived from LLM subcalls. These agents achieve up to 67% and 74% accuracy, respectively, exceeding observed expert performance. LLM-guided feature discovery generally outperforms direct reasoning, while Group Relative Policy Optimization (GRPO) improves open-weight models, enabling them to match or surpass closed-source frontier models in our evaluations. These results establish Pyrite as a challenging test of pose-ranking ability and highlight LLM-guided feature discovery as a promising approach to interpret and combine the outputs of computational chemistry tools.

Figures & tables

Figure 1: Interface for human expert annotations on our pairwise pose ranking benchmark Pyrite (ground truth crystal shown only for paper visualization). The example shows two predicted poses of a potential small-molecule inhibitor of the autotaxin protein ( Protein Data Bank 7G63 ), with toggle control for 2D contact map and interactive 3D and overlay views. Experts are asked to select which pose is closer to the crystallographic binding mode (selecting 1-5), write factors in favor of both pose A and B, and describe their reasoning on how they weigh these factors for pose selection. Annotators are also provided MM-GBSA scores and the # of unusual torsions using Mogul for each pose.
Figure 2: Top-k pose enrichment and diversity across cofolding models For each of Boltz2, Protenix, ESMFold2, and our cofolding model, we scored 120 poses per complex against the ground truth crystal, calling a pose a "hit" at RMSD < 2 Å and lDDT-PLI ≥\geq 0.8. Top Row shows top-k enrichment (EF@k = hits in the top-k over hits expected at random, pooled across complexes, so random = 1). Middle Row and Bottom Row : the 3rd-party cofolding models are far less diverse than ours, measured both by the distribution of per-pair RMSD and Interaction IoU.
PredictorPre-AF3 curatedPost-AF3 curatedPost-AF3 full
Human experts (abstain-as-wrong)0.42±0.030.42\pm 0.030.45±0.040.45\pm 0.04-
Human experts (decisive only)0.51±0.030.51\pm 0.030.57±0.050.57\pm 0.05-
MM-GBSA energy ↓\downarrow0.32±0.040.32\pm 0.040.34±0.060.34\pm 0.060.35±0.040.35\pm 0.04
Total interactions ↑\uparrow0.41±0.040.41\pm 0.040.40±0.060.40\pm 0.060.40±0.040.40\pm 0.04
Total torsion strain ↓\downarrow0.45±0.040.45\pm 0.040.51±0.060.51\pm 0.060.51±0.040.51\pm 0.04
Mogul unusual torsions ↓\downarrow0.35±0.040.35\pm 0.040.50±0.060.50\pm 0.060.43±0.040.43\pm 0.04
Table 1: Accuracy and 95% CIs from expert chemists, Boltz-2 confidence, and various physics scores from PyriteRep across three benchmark splits. No method achieves pairwise accuracy significantly above chance.
Figure 3: Human experts vs. models performance on post-AF3 curated split. (A) Accuracy for experts and a subset of ranking methods, split by Easy and Hard pairs. Error bars are 95% CIs. (B) Average accuracy across experts and two PyriteReasoner agents grouped by factors driving the reasoning for a given pair selection. (C) Rate at which experts and and two PyriteReasoner agents reference protein-family and domain knowledge in their reasoning output.
VariantOpus 4.8Sonnet 5GPT-5.6-solQwen3.8-27BQwen3.8-27B-GRPOΔ\Delta GRPO
Curated ( nn =265)
Raw 3D Coord Only0.49±0.060.49\pm 0.060.34±0.070.34\pm 0.070.52±0.060.52\pm 0.06 †0.53±0.060.53\pm 0.060.52±0.060.52\pm 0.06 †−0.01-0.01
Interactions + Raw 3D0.51±0.060.51\pm 0.06 †0.49±0.060.49\pm 0.060.55±0.060.55\pm 0.06 †0.58±0.060.58\pm 0.06 †0.62±0.060.62\pm 0.06 †+0.04+0.04
PyriteRep0.63±0.060.63\pm 0.06 †0.62±0.060.62\pm 0.06 †0.60±0.060.60\pm 0.06 †0.65±0.060.65\pm 0.06 †0.67±0.060.67\pm 0.06 †+0.02+0.02
PyriteRep w/o Raw 3D0.65±0.060.65\pm 0.06 †0.61±0.060.61\pm 0.06 †0.57±0.060.57\pm 0.06 †0.62±0.060.62\pm 0.06 †0.67±0.060.67\pm 0.06 †+0.05∗+0.05^{*}
In-Context0.62±0.060.62\pm 0.06 †0.58±0.060.58\pm 0.06 †0.47±0.060.47\pm 0.060.59±0.060.59\pm 0.06 †––
Table 2: PyriteReasoner pose ranking accuracy across four different base LLMs, five input ablations, and Qwen3.8-27B finetuned with GRPO. Values are mean accuracy ±\pm 95% confidence intervals. †\dagger means significantly higher than expert, curated split only, and * means significant difference for Δ\Delta GRPO, p<0.05p<0.05 .
ArmOpus 4.8Sonnet 5GPT-5.6-solQwen3.8-27BQwen3.8-27B-GRPOΔ\Delta
Curated (n=265), †\dagger means significantly higher than expert
Code Only0.65±0.060.65\pm 0.06 †0.63±0.060.63\pm 0.06 †0.65±0.060.65\pm 0.06 †0.52±0.060.52\pm 0.06 †0.69±0.060.69\pm 0.06 †+0.17∗+0.17^{*}
Protein LLMSubcall0.69±0.060.69\pm 0.06 †0.72±0.050.72\pm 0.05 †0.66±0.060.66\pm 0.06 †0.55±0.060.55\pm 0.06 †0.70±0.060.70\pm 0.06 †+0.15∗+0.15^{*}
Pose LLMSubcall0.69±0.060.69\pm 0.06 †0.73±0.060.73\pm 0.06 †0.71±0.060.71\pm 0.06 †0.59±0.060.59\pm 0.06 †0.74±0.050.74\pm 0.05 †+0.15∗+0.15^{*}
Vanilla Logistic Regression (all features in PyriteRep): 0.64±0.060.64\pm 0.06 †
All (n=484)
Table 3: PyriteFeaturizer pose ranking accuracy across four different base LLMs, give input ablations, and Qwen3.8-27B finetuned with GRPO. Values are mean accuracy ±\pm 95% confidence intervals. †\dagger means significantly higher than expert, curated split only, and * means significant difference for Δ\Delta GRPO, p<0.05p<0.05 .
Figure 4: GRPO training dynamics for PyriteReasoner (A) Train accuracy for four ablations across training steps (error bars are ±SEM) (B) Post-AF3 per-pair accuracy for four ablations across training steps in comparison with human experts (C) Post-AF3 per-pair accuracy for only the PyriteRep w/o 3D coordinates ablation, split by whether MM-GBSA picks incorrect pose. (D) Post-AF3 per-pair accuracy for only the PyriteRep w/o 3D coordinates ablation, split by whether Mogul # Unusual Torsions picks incorrect pose. Bottom Rows Example reasoning outputs from PyriteRep w/o 3D coordinates ablation over the course GRPO steps.
Figure 5: GRPO training dynamics for PyriteFeaturizer (A) Train mean-OOF accuracy for three ablations across training steps (error bars are ±SEM) (B) Post-AF3 per-pair accuracy for three ablations across training steps in comparison with human experts and vanilla logistic regression (C) Average # of distinct features in each program output for all three ablations across training steps (D) % of program outputs using gating logic on strain metrics for all three ablations across training steps. Bottom Rows Example features discovered by PyriteFeaturizer to use for logistic regression.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Methodpost-AF3 curatedpost-AF3 full
Logistic Regression0.64±0.060.64\pm 0.060.60±0.040.60\pm 0.04
Random Forest0.61±0.060.61\pm 0.060.65±0.040.65\pm 0.04
XGBoost0.59±0.060.59\pm 0.060.62±0.040.62\pm 0.04
LightGBM0.62±0.060.62\pm 0.060.62±0.040.62\pm 0.04
Appendix
Table 4: Pose-ranking accuracy of classical machine learning methods trained on quantitative features from PyriteRep . Values are mean accuracy ±\pm 95% confidence intervals on the post-AF3 curated and post-AF3 full evaluation splits.
ParameterPyriteReasonerPyriteFeaturizer
Model
Base modelQwen3.8-27B (decoder-only)Qwen3.8-27B (decoder-only)
LoRA rank / alpha16 / 3216 / 32
LoRA dropout0.00.0
LoRA targetsq/k/v/o, gate/up/down projq/k/v/o, gate/up/down proj
Trainable parameters79.7M / 27B (0.29%)79.7M / 27B (0.29%)
Appendix
Table 5: GRPO training hyperparameters for PyriteReasoner and PyriteFeaturizer (Qwen3.8-27B).
MethodAccuracyAgreement with sign-flipped MM-GBSA
Sign-flipped MM-GBSA64.7%100.0%
PyriteReasoner , zero-shot63.0%46.3%
PyriteReasoner , GRPO step 7067.4%51.4%
Appendix
Table 6: Accuracy on the post-AF3 full split and prediction-level agreement with the sign-flipped MM-GBSA rule. PyriteReasoner surpasses the sign-flipped rule while agreeing with it at close to the chance rate.
More in this topic

Learning to Rank

Learning to rank develops models that order items by relevance or utility for search, recommendations, and other decisions.

See all 289 papers

Explore similar work

CardsList

  1. Natural-Language-Guided Generator-Agnostic Shortlisting for Protein Binder Design

    Aug 21, 2026Gyubok Lee, Kiwoong Yoo, Jimin Seo +3LLM RerankingBioinformatics

    Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · LG AI Research, Seoul, South Korea · Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea +1

  2. RangeFinder: Predicting Ligand Property Ranges from Protein Pockets

    Nov 2, 2026Alan Nadelsticher Ruvalcaba, Chris Alvin, Supratik Mukhopadhyay +1Structure-Based Drug DesignMolecular Property Prediction

    Georgia Institute of Technology · Furman University · Louisiana State University

  3. An Interpretable Framework Applying Protein Words to Predict Protein-Small Molecule Complementary Pairing Rules

    Apr 17, 2026Jingke Chen, Jingrui Zhong, Tazneen Hossain Tani +3Protein-Ligand Binding Affinity PredictionInterpretable ML

    MOE Key Laboratory of Bioinformatics, State Key Laboratory of Molecular Oncology, Beijing Frontier Research Center for Biological Structure, School of Pharmaceutical Sciences, Tsinghua University, Beijing, 100084, China