cs.LGNov 18, 2026

Choosing the Right Representations: Task-Aware Fusion for Multimodal Materials Language Models

Authors: Huu Binh Ta, Alex Morehead, N. Benjamin Erichson, Ferdinando Fioretto, Sandeep Madireddy

Organizations: University of Virginia · Lawrence Berkeley National Laboratory · Argonne National Laboratory

Abstract

Multimodal language models for materials science rely on effective mechanisms for incorporating three-dimensional structural representations. MatterChat addresses this challenge using a large bridge module between a pretrained structural encoder and the LLM, together with a separate contrastive alignment stage. We instead ask whether this bridge can adaptively select and combine representations from multiple structural encoders according to the target property. We find that supervised and generative material encoders provide complementary information for property prediction. We found that, for Zatom-1, this information is not concentrated in the final layer. Instead, the most informative representations occur at different depths depending on the target property. This motivates Task-Conditioned Attention Fusion (TCAF), a lightweight attention-based bridge that uses task-conditioned latent queries to combine CHGNet features with intermediate Zatom-1 representations. The bridge is trained directly on downstream prediction tasks, allowing representation selection and material-language alignment to be learned jointly without separate contrastive pretraining. Across classification and regression tasks, TCAF outperforms MatterChat, with up to 47.8% lower RMSE, while using substantially fewer trainable parameters and halving the overall training time. Further experiments incorporating UMA-S demonstrate TCAF's flexibility across encoder combinations, with the three-encoder configuration achieving the highest macro-average classification accuracy of 89.61%.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Zatom-1: Towards a Multimodal Foundation Model for 3D Molecules and Materials

    Feb 24, 2026Alex Morehead, Miruna Cretu, Antonia Panescu +143D Foundation ModelsMultimodal Foundation Models

  2. Atomistic Language Models Understand and Generate Materials

    Jun 19, 2026Sathya Edamadaka, Krithik Ramesh, Ju Li +1Crystal Structure PredictionLanguage Modeling

  3. MolFM-Lite: Multi-Modal Molecular Property Prediction with Conformer Ensemble Attention and Cross-Modal Fusion

    Feb 25, 2026Syed Omer Shah, Mohammed Maqsood Ahmed, Danish Mohiuddin Mohammed +2Molecular Property PredictionMultimodal Fusion