QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery

Characterizing two-dimensional (2D) quantum materials by optical microscopy requires localizing exfoliated flakes and determining their layer thickness from subtle optical contrast and interference color to select suitable flakes for device fabrication. However, models face synthetic-to-real domain shifts and variation across materials, substrates, laboratories, and imaging conditions. We present QuPAINT, a physics-aware multimodal framework for transferable quantum flake characterization. The Synthetic Materials Framework (Synthia) generates diverse synthetic microscopy images while preserving layer-dependent optical behavior. Using these images, we construct QMat-Instruct, a multimodal instruction dataset with image-specific reasoning traces generated from verified annotations and constrained to observable optical cues. QuPAINT integrates these signals through Physics-Informed Attention (PIA), which injects substrate-relative optical priors into the visual representation to support grounded multimodal reasoning. For evaluation, we introduce QF-Bench, to our knowledge, the largest real-world benchmark for this problem, spanning diverse microscopy and substrate conditions. Using its verified annotations, we study counting, visual grounding, reasoning quality, confidence calibration, and transfer to an unseen material. QuPAINT-8B substantially outperforms prior methods and establishes state-of-the-art performance for both general and monolayer flake detection. Additional experiments show that image-grounded supervision improves strict spatial grounding and confidence calibration while preserving robust general flake detection on the unseen material.

Physics-aware multimodal large language models (MLLMs) can localize and characterize two-dimensional (2D) material flakes from optical microscopy images, producing rich candidate observations and scientific reasoning. However, downstream workflows also require reliable operations over these outputs, including counting, selection, measurement, and rendering. We present OpenQlaw, a tool-using agentic artificial intelligence (AI) system that connects a general vision-language orchestrator with Physics-Aware Instruction Tuning (QuPAINT), a physics-aware domain expert. Through evaluation on fully reviewed optical microscopy images, we show that summary counts derived from longer free-form candidate enumerations can become inconsistent with the underlying returned observations. Controlled replay experiments further show that structuring the same observations alone does not resolve this behavior, while deterministic delegation returns the exact requested record count in all cohort trials. To address this, we introduce State-Grounded OpenQlaw, which converts expert outputs into image-scoped observation records and performs deterministic scientific operations over explicitly identified records with refusal conditions when required information is unavailable. This preserves the expert's original analysis while making downstream operations exact over the selected record and independently evaluable from candidate-list discrepancies.

As multimodal language models play an increasingly important role in scientific research, materials science offers a critical testbed due to its interdisciplinary, multimodal, and application-driven nature. However, existing materials benchmarks mainly focus on property prediction, knowledge QA, or characterization understanding, leaving the broader reasoning process from materials knowledge to application underexplored. To fill this gap, we present OmniMatBench, a human-calibrated multimodal reasoning benchmark for materials science. OmniMatBench contains 3,171 expert-curated QA and calculation problems across 19 materials-science subfields, spanning fundamental materials knowledge, structural and engineering materials, materials processing and manufacturing, and functional and applied materials. We evaluate 13 open-source and closed-source MLLMs and find that the best model achieves only a 0.372 overall score, revealing a substantial gap in current materials-science reasoning. Further analysis shows strong variation across subfields, fixed reasoning heuristics, uneven materials knowledge, and limited high-level knowledge application under formula-, retrieval-, and code-assisted settings. OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs and establishes a foundation for reliable AI assistants in materials-science research.