When gene summaries reveal the answers: Gene Ontology circularity in gene embedding benchmarks
Organizations: Rutgers University · New York University
Abstract
A gene summary can contain the very annotations used to evaluate an embedding of its text. The Alliance of Genome Resources converts Gene Ontology (GO) annotations into sentences, which NCBI Gene displays and GenePT includes in its input. These summaries covered 6,172 of 19,256 human protein-coding genes (32.1%), including 58% of genes with one to ten linked publications. We embedded the same corpus with and without the generated summaries and refitted a function-prediction probe on each version. Among genes with one to ten publications, removal reduced the reported per-gene area under the receiver operating characteristic curve (AUROC) by 0.078 with a linear probe (95% bootstrap interval 0.072-0.084) and by 0.033 with a multilayer perceptron. These estimates use a historical GO scorer that mishandled ties. The original predictions are unavailable for rescoring. A simple TF-IDF encoder also benefited from the summaries, as did models evaluated only on experimental annotations. Replacing generated summaries with UniProt function descriptions recovered much of the lost performance despite little verbatim overlap. UniProt also curates many of the evaluation labels. In mouse and yeast, we found a similar dependence on the source of the description. We introduce GeneProv to compare encoders on matched text and to examine how representation rankings change across GO recovery, non-GO endpoints and co-essentiality.
Figures & tables
| Input | 0 | 1–10 | 11–100 | 101–1000 | 1000 | all |
|---|---|---|---|---|---|---|
| GenePT text, as found | 0.962 | 0.897 | 0.882 | 0.862 | 0.837 | 0.883 [0.930] |
| GenePT text, generated summaries removed | 0.935 | 0.819 | 0.855 | 0.859 | 0.843 | 0.849 [0.920] |
| GenePT text, replaced by LLM-written text † | 0.929 | 0.812 | 0.852 | 0.859 | 0.840 | 0.846 [0.909] |
| Gene symbol only | 0.936 | 0.843 | 0.844 | 0.848 | 0.849 | 0.848 |
| Rich text, as found | 0.989 | 0.954 | 0.923 | 0.893 | 0.867 | 0.926 [0.956] |
| Rich text, generated summaries removed | 0.941 | 0.912 | 0.913 | 0.890 | 0.865 | 0.908 [0.950] |
| Component | Definition | Motivation |
|---|---|---|
| Universe, strata | 19,256 HGNC protein-coding genes; five folds shared by every row; headline stratum genes with 0–10 publications (7,120), all genes beside it | exposure dependence (Section 2.2 ) |
| T1, GO recovery | 4,949 GO terms (no IEA or ND); Track F removes generated summaries; Track O uses declared inputs; per-gene AUROC, linear and MLP probes | generated summaries (Sections 2.1 – 2.5 ); still coupled to UniProt curation (Section 2.3 ) |
| T2, non-GO labels | eleven assay, population and curated endpoints (DepMap essentiality, gnomAD constraint, ClinGen dosage, six HPA locations, GTEx and HPA tissue specificity; the essentiality label and T4 share the DepMap source; ClinGen drops out of the 0–10 stratum by the class-size rule); folds grouped by HGNC family | labels outside GO (Section 2.5 ); homology between family members |
| T3, annotation holdout | experimental annotations added after 1 January 2023 to genes without experimental annotation then (such genes may carry inferred annotations; 137 genes with 0–5 publications at the cutoff); only representations eligible under the annotation-access rule, which admits later sequence/expression models (Methods) | temporal holdouts (Section 2.7 ) |
| T4, co-essentiality | retrieval of each gene’s ten DepMap co-essential partners by embedding similarity; no probe | DepMap-derived endpoint |
| GeneProv Index | mean over T1, T2 and T4 of ; T3 beside it; paired bootstrap intervals for every index and rank; transform independent of the other rows | declared task aggregation; task-level scores retained |
| Index | Genes with 0–10 publications | All genes | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # (95%) | Representation | text | 0–10 (95%) | all | T1 | T2 | T4 | T3 | T1 | T2 | T4 | T3 |
| 1 (1–2) | GB.Protein-16B rich summary-filtered text ‡ | mix | 47.1 (46.3–47.8) | 50.4 | 0.921 | 0.658 | 0.627 | – | 0.908 | 0.720 | 0.628 | – |
| 2 (1–2) | ProTrek-650M (text-aligned) | train | 47.0 (46.2–47.7) | 48.8 | 0.904 | 0.698 | 0.603 | – | 0.891 | 0.749 | 0.592 | – |
| 3 (2–3) | ProtST-ESM1b (text-aligned) | train | 46.3 (45.6–47.0) | 48.0 | 0.892 | 0.729 | 0.574 | 0.836 | 0.878 | 0.773 | 0.570 | 0.865 |
| 4 | GB.Protein-16B | – | 45.3 (44.5–46.0) | 46.2 | 0.875 | 0.676 | 0.627 | 0.780 | 0.857 | 0.719 | 0.616 | 0.833 |
| 5 (5–6) | ESM2-650M | – | 44.4 (43.7–45.1) | 44.9 | 0.879 | 0.690 | 0.597 | 0.777 | 0.860 | 0.729 | 0.585 | 0.839 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Input | Probe | 0 (152) | 1–10 (2,750) | 11–100 (4,798) | 101–1000 (4,618) | 1000 (2,789) |
|---|---|---|---|---|---|---|
| As found | linear | 0.949 | 0.871 | 0.818 | 0.781 | 0.717 |
| Generated removed | linear | 0.922 | 0.780 | 0.798 | 0.784 | 0.724 |
| Replaced by LLM text | linear | 0.929 | 0.786 | 0.797 | 0.782 | 0.716 |
| As found | MLP | 0.968 | 0.908 | 0.868 | 0.847 | 0.792 |
| Generated removed | MLP | 0.941 | 0.832 | 0.847 | 0.846 | 0.789 |
| Replaced by LLM text | MLP | 0.950 | 0.829 | 0.839 | 0.844 | 0.787 |
| Input | 0 | 1–10 | 11–100 | 101–1000 | 1000 | all | per term |
|---|---|---|---|---|---|---|---|
| GenePT text, as found | 0.986 | 0.952 | 0.928 | 0.903 | 0.871 | 0.930 | 0.902 |
| GenePT text, generated summaries removed | 0.972 | 0.918 | 0.923 | 0.912 | 0.881 | 0.920 | 0.872 |
| GenePT text, replaced by LLM-written text † | 0.968 | 0.906 | 0.910 | 0.903 | 0.869 | 0.909 | 0.876 |
| Rich text, as found | 0.993 | 0.978 | 0.957 | 0.929 | 0.895 | 0.956 | 0.932 |
| Rich text, generated summaries removed | 0.982 | 0.965 | 0.952 | 0.926 | 0.894 | 0.950 | 0.933 |
| Rich text, removed and gene names masked | 0.930 | 0.948 | 0.950 | 0.926 | 0.892 | 0.943 | 0.905 |
| Genes / bin | As found | UniProt text | Removed | As found UniProt | As found removed | |
|---|---|---|---|---|---|---|
| Generated, 0 | 95 | 0.948 | 0.887 | 0.887 | (0.032 to 0.095) | |
| Generated, 1–10 | 1,938 | 0.909 | 0.905 | 0.756 | ( to 0.012) | |
| Generated, 11–100 | 1,712 | 0.902 | 0.895 | 0.774 | ( to 0.014) | |
| Generated, 101–1000 | 197 | 0.874 | 0.854 | 0.789 | (0.001 to 0.041) | |
| RefSeq, 1–10 | 1,353 | 0.901 | 0.933 | ( to ) | ||
| RefSeq, 11–100 | 5,763 | 0.881 | 0.911 | ( to ) |
| Organism | Genes | All | 0 | 1–10 | 11–100 | 101–1000 | RefSeq summary |
|---|---|---|---|---|---|---|---|
| Human | 20,606 | 30.0 | 29.0 | 56.6 | 22.1 | 6.4 | 58.2 |
| Mouse | 26,365 | 69.6 | 32.7 | 90.0 | 89.6 | 81.6 | 10.0 |
| Rat | 25,140 | 75.8 | 42.7 | 97.6 | 97.6 | 95.2 | 5.5 |
| Zebrafish | 34,930 | 61.5 | 39.2 | 96.4 | 99.0 | 98.2 | 0.2 |
| Fly | 13,986 | 86.0 | 59.2 | 92.6 | 99.7 | 100.0 | 0.0 |
| Worm | 19,983 | 63.8 | 47.5 | 93.9 | 99.9 | 100.0 | 0.0 |
| Index | Genes with 0–10 publications | All genes | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # (95%) | Representation | text | 0–10 | all | T1 | T2 | T4 | T3 | T1 | T2 | T4 | T3 |
| 1 | GB.Protein-16B rich summary-filtered text ‡ | mix | 58.2 | 59.5 | 0.970 | 0.777 | 0.627 | – | 0.952 | 0.812 | 0.628 | – |
| 2 (2–3) | GB.Protein-16B | – | 54.3 | 53.6 | 0.945 | 0.743 | 0.627 | 0.917 | 0.926 | 0.763 | 0.616 | 0.928 |
| 3 (2–3) | ProTrek-650M (text-aligned) | train | 53.9 | 54.0 | 0.960 | 0.747 | 0.603 | – | 0.943 | 0.775 | 0.592 | – |
| 4 | ESM1b | – | 53.2 | 52.2 | 0.945 | 0.742 | 0.612 | 0.916 | 0.928 | 0.755 | 0.601 | 0.928 |
| 5 (5–7) | ESM2-650M | – | 52.1 | 51.4 | 0.946 | 0.739 | 0.597 | 0.914 | 0.929 | 0.757 | 0.585 | 0.930 |
| Genes with 0–10 publications | All genes | ||||||||
| Representation | version | Index (95%) [MLP] | T1 | T2 | T4 | Index [MLP] | T1 | T2 | T4 |
| GB.Protein-16B | v1 | 45.3 (44.5–46.0) [54.3] | 0.875 | 0.676 | 0.627 | 46.2 [53.6] | 0.857 | 0.719 | 0.616 |
| ESM2-650M | v1 | 43.6 (42.9–44.4) [51.1] | 0.872 | 0.690 | 0.592 | 44.1 [50.0] | 0.852 | 0.728 | 0.582 |
| Rich summary-filtered text | v1 | 42.8 (42.0–43.6) [52.1] | 0.915 | 0.651 | 0.576 | 47.8 [55.1] | 0.908 | 0.722 | 0.587 |
| UniProt function text only | v1 | 42.5 (41.7–43.2) [51.6] | 0.916 | 0.647 | 0.574 | 46.6 [54.0] | 0.907 | 0.706 | 0.585 |
| GB.Protein-16B rich summary-filtered text | v1 | 47.1 (46.3–47.8) [58.2] | 0.921 | 0.658 | 0.627 | 50.4 [59.5] | 0.908 | 0.720 | 0.628 |
| Probe | Labels | Bin | As found | Removed | As found removed (95%) | |
|---|---|---|---|---|---|---|
| linear | standard | 0 | 568 | 0.962 | 0.935 | (+0.014 to +0.040) |
| IBA excluded | 0 | 100 | 0.861 | 0.772 | (+0.031 to +0.145) | |
| standard, fixed set | 0 | 90 | 0.871 | 0.780 | (+0.039 to +0.144) | |
| standard | 1-10 | 4,310 | 0.897 | 0.819 | (+0.072 to +0.084) | |
| IBA excluded | 1-10 | 3,511 | 0.877 | 0.791 | (+0.079 to +0.093) | |
| standard, fixed set | 1-10 | 3,405 | 0.883 | 0.797 | (+0.078 to +0.092) |
| Representation | Index | Gene CI | Family CI | Gene rank | Family rank | T2, fixed | Index, fixed | |||
|---|---|---|---|---|---|---|---|---|---|---|
| GB.Protein-16B rich summary-filtered text | 47.1 | 46.3–47.8 | 41.8–51.4 | 1 (1–2) | 1 (1–3) | 7 | 2 | 1 | 0.723 (0.720) | 50.6 (50.4) |
| ProTrek-650M | 47.0 | 46.2–47.7 | 41.3–51.7 | 2 (1–2) | 2 (1–3) | 5 | 1 | 2 | 0.759 (0.749) | 49.5 (48.8) |
| ProtST-ESM1b | 46.3 | 45.6–47.0 | 42.4–50.0 | 3 (2–3) | 3 (1–4) | 4 | 3 | 3 | 0.791 (0.773) | 49.2 (48.0) |
| GB.Protein-16B | 45.3 | 44.5–46.0 | 40.1–49.9 | 4 (4–4) | 4 (3–4) | 3 | 4 | 6 | 0.735 (0.719) | 47.3 (46.2) |
| ESM1b | 43.9 | 43.1–44.7 | 39.1–48.4 | 5 (5–6) | 5 (5–6) | 8 | 5 | 8 | 0.729 (0.714) | 45.4 (44.5) |
| ESM2-650M | 43.6 | 42.9–44.4 | 38.9–48.0 | 6 (5–7) | 6 (5–8) | 9 | 6 | 10 | 0.745 (0.728) | 45.2 (44.1) |
| Missing by publication count | 0–10 papers | All genes | |||||||||
| Representation | Present | Missing | 0 | 1-10 | 11-100 | 101-1000 | AUROC | R@10 | AUROC | R@10 | |
| GB.Protein-16B | 19,133 | 123 | 16 | 59 | 42 | 6 | 0 | 0.627 | 0.017 | 0.616 | 0.016 |
| ESM2-650M | 19,133 | 123 | 16 | 59 | 42 | 6 | 0 | 0.592 | 0.015 | 0.582 | 0.013 |
| GB.Cell-100M | 18,593 | 663 | 295 | 277 | 71 | 19 | 1 | 0.580 | 0.005 | 0.578 | 0.008 |
| Geneformer V2-316M | 19,229 | 27 | 4 | 11 | 10 | 2 | 0 | 0.588 | 0.004 | 0.590 | 0.004 |
| GenePT ada-002 (as released) | 18,942 | 314 | 85 | 168 | 46 | 13 | 2 | 0.562 | 0.017 | 0.565 | 0.022 |
| Representation | Training data (as stated by the authors) | Data date | Annotation or text signal in training | T3 eligible | Coverage |
|---|---|---|---|---|---|
| Sequence and expression models (Track O) | |||||
| GB.Protein-16B [ 25 ] | 1.2 trillion amino acids from UniRef90 and ColabFoldDB; masked language modelling | released Nov 2024; database releases not stated | none (sequences only) | yes, by the sequence rule | 19,133 |
| ESM2-650M [ 26 ] | UniRef50 clusters sampled from UniRef90 (UniRef release 2021_04), masked language modelling | 2021 | none | yes | 19,133 |
| ESM1b [ 30 ] | UniRef50 2018_03, masked language modelling | 2018 | none | yes | 19,133 |
| GB.Cell-100M [ 39 ] | 50 million single-cell transcriptomes from over 100 tissue types, read-depth-aware masked modelling | released Nov 2024 | none (expression counts) | yes, by the expression rule | 18,593 |
| Geneformer V2-316M [ 42 ] | about 104 million human single-cell transcriptomes (Genecorpus-104M), rank-value encoding | trained Dec 2024 | none | yes | 19,229 |
| Round | Content | Plan written (UTC) | Plan committed | Results committed | Added after the plan (post hoc) | Gates |
|---|---|---|---|---|---|---|
| 7 | the circularity audit under a documented analysis plan: prevalence, inflation, label sets, encoders, published embeddings and benchmarks, temporal holdouts, time travel, assay labels (Sections 2.1 – 2.6 , 2.7 , 3.3 ) | 2026-10-09 03:57 (signed off by the author) | 2026-10-09 15:38, 561c9c2, with the results | same commit | protein model against frozen literature-only text; fixed-model assay comparison; text-versus-protein by exposure; GenePT’s 2023 text in the Zhong et al. temporal task (prediction wrong, logged) | 16 passed; 4 failed (D3: non-GO labels move; G1, G2: time-travel and fusion thresholds; S for ProtST) |
| 8 | LLM-written descriptions, scELMo, model organisms, UniProt statements inferred from sequence (Sections 2.8 , 2.9 , 3.3 ) | 2026-10-10 01:02 | 2026-10-10 01:43, 41d7aeb, with the results | same commit | protein model against LLM-written text by exposure; ESM2 in place of GB.Protein-16B; correction of the verbatim-overlap figures (summary-only text) | 7 passed; 1 failed (H2: time travel against frozen LLM text) |
| 9 | simulated review and revision; UniProt text in place of the summaries, nearest-neighbour probe, fold seeds, family-restricted time travel (Section 2.3 ) | 2026-10-10, before 05:19 | 2026-10-10 05:19, 3790283, before computation | 2026-10-10 15:12, df9542c | co-curation test (declared before it was computed); replacement by LLM text; per-term AUROC within bins; ESM2 on the zero-literature comparison; one detector phrase added after the planned evaluation | 3 passed (L, M, N); 2 failed (K1, K2: UniProt text recovers the loss) |
| 10 | GeneProv v0, the frozen benchmark table | 2026-10-10, before 15:37 | 2026-10-10 15:37, e4ebbd7, before computation | 2026-10-10 16:02, 8026128 | none | 4 passed (R, V1–V3) |
| 11 | GeneProv v1: the index, seven rows, Track F index, leaderboard | 2026-10-11, before 07:25 | 2026-10-11 07:25, e541de0, before computation | 2026-10-11 08:10, 779a1d5 | none; the first run of the table script applied gate I1 to the text-aligned encoders, contrary to the plan’s wording, and was corrected and rerun (logged 07:53, before the rerun) | 4 passed (R, I1–I3); predictions without gates held |
| 12, 13 | restructuring of the manuscript into two parts; close-out (author matter, deposit) | 2026-10-12 | 2026-10-12 06:40, 3a99920; 18:43, e67c476 | 06:52, 8052098; 18:48, d94d355 | no number computed | none |
Data Leakage
Data leakage is unintended information flow into model training, evaluation, or outputs that inflates results or exposes sensitive data.
See all 137 papers