cs.LGOct 21, 2026
SaveAI Mindmeld: A Latent Interconnect Protocol
Organizations: Polymathic AI · Center for Data Science, New York University · Department of Applied Physics, Yale University · Center for Computational Astrophysics, Flatiron Institute ·
Abstract
LLMs can act as scientific agents using tools to consult specialists and run experiments. Yet specialist information must enter their context, generally as text, images, audio or video: a bottleneck between models with rich, high-dimensional representations. We advocate direct representation exchange: a mindmeld conveying many properties, including information an LLM would not think to request. It should be extensible to new specialists, flexible across questions and tasks, and compositional across independently connected specialists without joint training. The Latent Interconnect Protocol (LIP) learns connectors mapping specialist representations into LLM internal states. Training only connectors keeps this practical as LLMs grow and evolve. We connect three protein and three small molecule specialists to Gemma 3 (4B and 12B parameters) and Kimi K2.6 (1T total, 32B active), freezing all pretrained models. Independently trained connectors provide useful information without changing the LLM or existing connectors, supporting familiar and new question types about unseen objects. We demonstrate zero-shot transfer to cross-domain tasks without joint training. Task performance and compositionality generally improve with LLM capability, with especially strong gains in zero-shot task transfer. Limitations remain: current LLMs were not trained for specialist representations as native inputs; access does not ensure reliable interpretation, integration or use in reasoning. We call on specialist developers to provide consistent representation interfaces and, especially, on LLM developers to train models to receive, combine and reason with these inputs. These efforts could expand scientific agents' information while preserving conventional tools' flexibility. Data, pretrained connectors, and code for training, evaluation and interactive inference: https://github.com/PolymathicAI/AI-Mindmeld
Figures & tables
Figure 1: Latent Interconnect Protocol alongside tool calling. (a) The tool-output route returns selected properties as words or numbers; the latent route supplies specialist representations through learned connectors. The question and outputs are illustrative, and one agent can use both routes. (b) Connectors selected from the registry write the protein and compound representations into separate prompt slots at the marked layers. The symbols denote the mixing operation defined in Section 2.1 . The layer choices are schematic and may differ between specialists. Each representation occupies one token here; LIP also permits multiple-token slots. Registry entries are trained independently, and new connectors can be added without changing the shared language model or existing connectors. The displayed models and domains illustrate the registry’s organization.
Figure 2: Three stages of independent connector training. 1. Align. Layer-specific maps match specialist representations to text-derived reference states in the LLM. The schematic illustrates alignment to description states; the connectors evaluated here instead align to states obtained from object names. 2. Select. Injection layers are selected to span the LLM’s depth, guided by local peaks in single-layer AUROC on held-out scan objects. 3. Train. The selected maps write into the object’s slot and are refined by answer loss. Gradients pass through the frozen LLM to the connector; both pretrained models remain frozen throughout. Layer counts, bar lengths and example text are illustrative, not measured results or recorded model responses. Table 2 gives the actual schedules.
Figure 3: Varied supervision through one connector. (a) Configured shares among non-variant examples. Illustrative prompt–target combinations, not logged draws or generated answers; the description answer is abbreviated and the relational pair is schematic. The full description field is truncated at 48 tokens during training. (b) Prompt variations in slot position, worked context, answer format and reasoning. All numbered slots within a prompt use the same connector. (c) Inner rings show the mix of training streams; outer rings show the question types. Protein training additionally includes questions about mutation effects, pathogenicity and loss of function. Percentages refer to the full training mix in each domain. Headers give training-pool sizes; sampling details are in Supplementary Section A.2.3 .
| Specialist | Representation | Dim. |
|---|---|---|
| Proteins | ||
| MIMIC | Layer 16 (mean pooled) Layer 20 (mean pooled) | 3,072 |
| ESM-C 600M | Layer 33 (mean pooled) Layer 36 (mean pooled) | 2,304 |
| PGLM-1B | Layer 24 (mean pooled) | 2,048 |
| Molecules | ||
| MoLFormer-c3 | Layer 9 (mean pooled) Layer 12 (EOS token) | 1,536 |
Table 1: Specialist read layers and representations. Read layers were chosen using property probes and, for proteins, variant probes; complementary layers were concatenated when they improved performance on at least one task.
Figure 4: Object, task and cross-domain evaluation design. (a) Training and held-out object pools, question types and data sources. Training counts give the pools used by the fitted connectors; held-out counts give the full validation and test splits, not evaluated sample sizes. The joint card reports evaluation groups from Rhea and BindingDB for reaction participation and protein–compound binding. Cross-domain evaluations include both familiar and held-out objects. (b) The progression from trained questions about held-out objects to held-out questions about held-out objects, then cross-domain questions using independently trained connectors. Solid slots denote training objects and dashed slots denote held-out objects. The example questions are illustrative. Within-domain composition of different connectors is an additional evaluation described in the text. Final task definitions, scored populations and controls are in Supplementary Section B .
| LLM | Proteins | Molecules |
|---|---|---|
| Gemma 3 4B | 2, 7, 11, 17, 26, 33 | 4, 7, 12, 16, 21, 32 |
| Gemma 3 12B | 6, 11, 16, 23, 33, 44 | 4, 9, 17, 23, 31, 41 |
| Kimi K2.6 | 7, 13, 22, 33, 49, 58 | 7, 13, 22, 32, 49, 57 |
Table 2: LLM injection layers. Shared layers for each domain and LLM were chosen to span its depth, guided by local peaks in the maximum AUROC envelope over connectors and the LLM’s own-state reference.
Figure 5: Within-task generalization to unseen objects. (a–b) Task-group AUROC for protein and molecule connectors attached to Kimi K2.6. Protein connectors perform similarly across groups, while molecule connectors differ most on heterocycles. (c) Mean AUROC across the three specialists in each domain for Gemma 3 4B, Gemma 3 12B and Kimi K2.6. Protein performance increases across these LLMs, whereas molecule performance varies less and peaks with Gemma 3 12B. Error bars in (c) show 95% bootstrap confidence intervals.
Figure 6: Zero-shot transfer to new tasks on unseen objects. (a–b) Task-group AUROC for protein and molecule connectors attached to Kimi K2.6. Protein connectors show similar transfer performance, while molecule transfer is strongest for mass. (c) Mean AUROC across the three specialists in each domain for Gemma 3 4B, Gemma 3 12B and Kimi K2.6. Performance increases across these LLMs in both domains. Error bars in (c) show 95% bootstrap confidence intervals. ∗ Starred question types had exposure up to 0.3%, and in some cases related information may have been implied in the training examples’ free-text descriptions.
Figure 7: Effects of connector training on trained and unseen tasks. Open and filled markers show AUROC after alignment and after connector training, respectively, averaged across the three specialists in each domain for each LLM. (a–b) Training improves performance on both trained and unseen protein tasks. (c) Training generally improves performance on trained molecule tasks. (d) Performance declines on most unseen molecule tasks; mass prediction improves only for Kimi K2.6. ∗ Starred question types had exposure up to 0.32%, and in some cases related information may have been implied in the training examples’ free-text descriptions.
Figure 8: Zero-shot composition of independently trained connectors. Compositionality improves with LLM capability, with Kimi K2.6 benefiting most from combining independently trained connectors. (a–b) Individual and combined protein connectors on trained and unseen tasks. (c) Rhea reaction-participation ordering. The first three columns compare LLMs using ESM-C 600M and ChemFM 1B with single-token linear connectors. The final column shows Kimi K2.6 using ESM-C 6B and ChemFM 3B with four-token MLP connectors (Section 4.5 ). Point estimates are higher for seen molecules. All proteins are unseen: nearly every seen-protein pair was excluded because the protein connector’s training descriptions already stated the queried relation. Error bars show 95% bootstrap confidence intervals.
Figure 9: LLM responses to latent inputs and the effect of reasoning. (a) Examples of the LLM treating injected protein embeddings as protein names. (b) Mean task AUROC for direct answers and answers after reasoning, averaged across protein connectors for each LLM. Reasoning reduces protein-property prediction performance for all three LLMs. Bars in (b) show 95% object bootstrap confidence intervals, using shared draws across tasks and connectors.
Figure 10: Latent inputs preserve useful distinctions beyond names. (a) Protein-property prediction using embeddings or names generated from those embeddings. Retaining the embedding gives higher AUROC for all three LLMs. (b–c) Lineage and length discrimination among proteins sharing the same name, measured by within-name concordance; lineage uses four low-exposure questions, and length tests whether a protein exceeds 320 amino acids. Embeddings provide information beyond the shared name, with Kimi K2.6 showing the strongest performance in both tests. Supplied names score 0.5 by construction. Error bars show 95% bootstrap confidence intervals.
Figure 11: Effects of write location and token count. Gemma 3 12B with MIMIC (a) and ChemFM-1B (b) . Mean of trained and unseen task performance; error bars show 95% bootstrap intervals. The paper’s connector uses one token and six injection layers. Moving a single-token write from the input to the best layer improves performance, supporting deep injection. Six input tokens match the six-layer connector’s parameter count but require more inference compute; even so, performance is higher than single-token injection at the best layer but below the six-layer connector.
Figure 12: Connector extensions. ESM-C 6B with Kimi K2.6: mean AUROC across trained and unseen tasks after 2,000 training steps. Error bars show 95% bootstrap intervals.
Figure 13: Specialist and LLM scaling. (a) Proteins: mean task AUROC and variant pair accuracy; (b) molecules: mean task AUROC. Protein property performance varies more across LLMs than across specialist sizes, while variant effects show the opposite pattern. Error bars show 95% confidence intervals, including seed variation for Gemma 3 4B and 12B.
Figure 14: Comparison with supervised linear probes. Kimi K2.6 results using the four-token MLP connector of the previous section. On trained tasks, connectors match logistic regression in both domains. On new tasks, connectors trail the supervised probes, but are evaluated zero-shot. Scores are mean task AUROC or variant pair accuracy; error bars show 95% bootstrap intervals.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Task supervision | Reserved question categories |
|---|---|---|
| Protein | Function, family, subunit, catalytic activity, EC class, pathway, location and selected keywords; relations and variant effects, pathogenicity and loss of function. | Lineage, length, selected modifications, cofactors and domains. |
| Compound | Definitions, mechanism roles, structural classes and functional parents; relations between compounds. | Selected provenance, application and chemical roles; formula, mass, charge and element-based questions. |
Table 3: Task supervision and reserved categories. Reserved questions may have related information in description targets.
Figure 15: Protein specialist read profiles. Mean AUROC of property (green) and variant (purple) probes for mean-pooled representations across specialist layers. Vertical lines mark chosen layers, and dashed lines show the corresponding scores of the chosen single-layer or concatenated representation.
Figure 16: Molecule specialist read profiles. Mean property-probe AUROC across specialist layers, using mean-pooled embeddings for MoLFormer, Uni-Mol2 and ChemFM 1B, and concatenated last-token and mean-pooled embeddings for ChemFM 3B. Vertical lines mark chosen layers, and dashed lines show the scores of the full chosen representations.
Figure 17: Name and description states as alignment targets. Mean task AUROC at each scanned layer in the informative band of Kimi K2.6, injecting (a) ridge-predicted states or (b) the LLM’s actual reference states.
Figure 18: LLM injection-layer profiles. Mean diagnostic-task AUROC from single-layer injection of aligned connector outputs (gray) and the LLM’s own reference states (dashed), with the maximum across the displayed curves in black and chosen layers marked vertically. Rows show proteins and molecules; columns show Gemma 3 4B, Gemma 3 12B and Kimi K2.6.
Figure 19: Complementary specialists improve information merging. Mean task AUROC with Kimi K2.6 on 2,036 proteins for (a) five trained and (b) eight unseen questions, comparing three different specialists with the best single connector and the best triple repeating one specialist. Intervals are 95% paired protein-bootstrap intervals (2,000 draws).
More in this topic
Large Language Model-Guided Scientific Discovery
Large language model-guided scientific discovery uses AI to propose, test, and refine hypotheses, models, experiments, and designs.
See all 278 papers