Pyrite: Building and Benchmarking Protein-Ligand Pose Ranking Agents
Organizations: Stanford University · Genesis Molecular AI
Abstract
Predicting how a ligand binds to its target is central to small-molecule drug discovery. Modern cofolding models can generate diverse candidate poses, but reliably ranking these candidates remains challenging. We introduce Pyrite, a pairwise pose-ranking benchmark comprising 1,500 pose pairs across diverse protein-ligand complexes. Expert chemists equipped with 3D visualization tools and physics-based scores perform near chance, highlighting the difficulty of interpreting available chemical signals. To investigate automated ranking, we introduce PyriteRep, a structured representation of molecular geometry, protein-ligand interactions, and physicochemical scores, and develop two LLM-based agents: PyriteReasoner, which directly compares poses through text-based reasoning, and PyriteFeaturizer, which generates executable code to construct features for logistic regression, optionally incorporating features derived from LLM subcalls. These agents achieve up to 67% and 74% accuracy, respectively, exceeding observed expert performance. LLM-guided feature discovery generally outperforms direct reasoning, while Group Relative Policy Optimization (GRPO) improves open-weight models, enabling them to match or surpass closed-source frontier models in our evaluations. These results establish Pyrite as a challenging test of pose-ranking ability and highlight LLM-guided feature discovery as a promising approach to interpret and combine the outputs of computational chemistry tools.
Figures & tables
| Predictor | Pre-AF3 curated | Post-AF3 curated | Post-AF3 full |
|---|---|---|---|
| Human experts (abstain-as-wrong) | - | ||
| Human experts (decisive only) | - | ||
| MM-GBSA energy | |||
| Total interactions | |||
| Total torsion strain | |||
| Mogul unusual torsions |
| Variant | Opus 4.8 | Sonnet 5 | GPT-5.6-sol | Qwen3.8-27B | Qwen3.8-27B-GRPO | GRPO |
|---|---|---|---|---|---|---|
| Curated ( =265) | ||||||
| Raw 3D Coord Only | † | † | ||||
| Interactions + Raw 3D | † | † | † | † | ||
| PyriteRep | † | † | † | † | † | |
| PyriteRep w/o Raw 3D | † | † | † | † | † | |
| In-Context | † | † | † | – | – | |
| Arm | Opus 4.8 | Sonnet 5 | GPT-5.6-sol | Qwen3.8-27B | Qwen3.8-27B-GRPO | |
|---|---|---|---|---|---|---|
| Curated (n=265), means significantly higher than expert | ||||||
| Code Only | † | † | † | † | † | |
| Protein LLMSubcall | † | † | † | † | † | |
| Pose LLMSubcall | † | † | † | † | † | |
| Vanilla Logistic Regression (all features in PyriteRep): † | ||||||
| All (n=484) | ||||||
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | post-AF3 curated | post-AF3 full |
|---|---|---|
| Logistic Regression | ||
| Random Forest | ||
| XGBoost | ||
| LightGBM |
| Parameter | PyriteReasoner | PyriteFeaturizer |
| Model | ||
| Base model | Qwen3.8-27B (decoder-only) | Qwen3.8-27B (decoder-only) |
| LoRA rank / alpha | 16 / 32 | 16 / 32 |
| LoRA dropout | 0.0 | 0.0 |
| LoRA targets | q/k/v/o, gate/up/down proj | q/k/v/o, gate/up/down proj |
| Trainable parameters | 79.7M / 27B (0.29%) | 79.7M / 27B (0.29%) |
| Method | Accuracy | Agreement with sign-flipped MM-GBSA |
|---|---|---|
| Sign-flipped MM-GBSA | 64.7% | 100.0% |
| PyriteReasoner , zero-shot | 63.0% | 46.3% |
| PyriteReasoner , GRPO step 70 | 67.4% | 51.4% |
Learning to Rank
Learning to rank develops models that order items by relevance or utility for search, recommendations, and other decisions.
See all 289 papers