PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning

Kecheng Liang; Haoyang Liu; Zexin Chen; Zirong Liu; Weixing Chen; Qiufeng Wang; Yang Liu; Liang Lin

cs.AISep 27, 2026

PhysAlign: A Benchmark for Evidence-Grounded Role Alignment in Multimodal Physics Reasoning

Authors: Kecheng Liang, Haoyang Liu, Zexin Chen, Zirong Liu, Weixing Chen, Qiufeng Wang, Yang Liu, Liang Lin

Organizations: Sun Yat-sen University · Dalian University of Technology · Xi’an Jiaotong-Liverpool University

Abstract

A key challenge in physics diagram understanding is correctly associating visual information with the physical entities, relations, and conditions it describes. Even when a value, symbol, or other local element is accurately recognized, assigning it to the wrong entity or scope can distort the underlying physical premise and lead to incorrect reasoning. To systematically study this challenge, we introduce \textbf{PhysAlign}, a benchmark designed to assess whether multimodal models correctly associate information recognized from physics diagrams with its intended physical role. By disentangling visual recognition from physical-role assignment through localized probes and controlled variants, PhysAlign isolates correspondence errors from recognition failures. It contains 3,341 human-validated probes spanning 986 physics problems, enabling systematic evaluation of visual recognition and physical-role correspondence at scale. We further introduce five complementary evaluation metrics, including CAcc, GAcc, and JAcc, which provide a comprehensive assessment of models' ability to recognize diagram content, establish correct physical correspondences, and solve the underlying physics problem. Across our evaluated multimodal models, PhysAlign reveals a consistent gap between local visual recognition and physical-role grounding. Even when the queried content is correctly recognized, the conditional correspondence error rate remains 13.8% for GPT-6-Astra and rises to about 50.6% for InternVL3.5-8B. These findings indicate that strong perception alone does not ensure reliable physical interpretation, exposing a distinct grounding bottleneck that is largely hidden by answer-level accuracy and highlighting the need for future models to better align recognized visual evidence with its physical meaning.

Figures & tables

Figure 1: Correct recognition does not guarantee physical-role grounding. (a) The problem evidence specifies a rod supported by a cable, with R1 marking the symbol

T

. (b) The reference associates

T

with the supporting cable ( E4 ), whereas the illustrated response assigns it to the rod ( E3 ). (c) Recognition is correct (

C_{j}=1

), but grounding and joint correctness fail (

G_{j}=0

J_{j}=0

). Reference targets and scores are used only for evaluation and are not provided to the model.

Figure 2: Data construction and quality assurance in PhysAlign. Standardized source problems undergo answer-blind annotation (Passes 1–4), human review, and probe compilation. Pass 5 aligns reference solutions with evidence for analysis only and does not supply probe targets. Paired base/+GT inputs share the original problem, evidence anchors, questions, and candidates; +GT supplies correct local readings without disclosing the queried role target. Structural checks and instance audits precede packaging, which separates public inputs from private targets and scoring mappings.

Figure 3: Domain coverage and representative physics problems in PhysAlign. Left: Distributions of source problems, localized probes, and paired +GT inputs across seven physics domains and a cross-disciplinary group. Each radar series shows percentages within its corresponding collection; counts beside each group follow the same order. Right: Representative question–diagram examples from the seven domains, illustrating an educational span from middle school to university-level physics.

(a) Source coverage
Source dataset	Parents	Parents [-2pt] with +GT
SeePhys ( Xiang et al., 2025 )	268	119
LiveK12Bench ( Wang et al., 2026 )	250	88
PhysElite ( Xu et al., 2026 )	220	79
OlympiadBench a ( He et al., 2024 )	209	68
Gaokao-MM-Physics ( Zong and Qiu, 2024 )	23	17

Table 1: Source coverage and release partition of PhysAlign.

Figure 4: Recognition–grounding decomposition and paired controls. (a) Weighted outcome shares on $\mathcal{L}$ ( $N=311$ , $n=412$ ). (b) Base/+GT grounding and paired changes with 95% confidence intervals on $\mathcal{P}_{m}$ ; model-specific coverage is shown above the panel (Appendix B.5 ). $N$ / $n$ count parents/probes. +GT supplies local readings, not perfect perception.

Rule-based baselines ( $GAcc\,\uparrow$ )
Random candidates: 18.69% Nearest region: 76.82%
Information controls and interface robustness
	Reference	No image		Reordered
Model	$GAcc\,\uparrow$	$GAcc\,\uparrow$	$\Delta G$ (pp)	$GAcc\,\uparrow$
Qwen3.5-4B ( Qwen Team, 2026 )	40.21	38.78	$-1.43$	41.93
Qwen3.5-9B ( Qwen Team, 2026 )	40.52	43.53	$+3.01$	43.02

Table 3: Grounding validity and robustness: rule-based baselines, image removal, and candidate reordering.

Figure 5: Local grounding by binary final-answer correctness, not by partial-credit

SolveAcc

. (a) Parent-level means for solve-wrong and solve-correct groups (

N=311

parents,

n=412

probes). (b) Correct-minus-wrong difference in percentage points. Estimates use independent solving and probing runs.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Recorded status or outcome	Probes
(a) Initial model-screening states
Approved by model screening	3,370
Rejected by model screening	1,128
Further review required at initial screening	1,089
Incomplete / call error	7
Total generated candidates	5,594

Appendix

Table 4: Candidate-probe screening, release admission, and human re-review. Panel (a) records the initial model-screening states of 5,594 generated probes; panel (b) reports admission within the model-approved subset. Panel (c) reports final human-review outcomes for revised versions of the 1,089 candidates initially requiring further review.

(a) Sampling allocation
Audit source group a	Total probes	Audited	Coverage (%)
Local	1,679	1,176	70.04
PhysElite	792	554	69.95
OlympiadBench	669	469	70.10
Expansion-v3	123	123	100.00
Phyx-OE	40	40	100.00

Appendix

Table 5: Post-construction quality audit of retained probes. (a) Sampling allocation. (b) Review outcomes within the audited set.

	All parents		Paired-input coverage
Domain group	Count	Share (%)	Paired	Unpaired	Rate (%)
Mechanics a	478	48.48	208	270	43.51
Electromagnetism a	267	27.08	112	155	41.95
Optics	50	5.07	15	35	30.00
Thermal physics / thermodynamics	48	4.87	10	38	20.83
Waves and acoustics	35	3.55	9	26	25.71

Appendix

Table 6: Estimated domain composition and parent-level paired-input coverage of PhysAlign. Counts refer to parent problems; domain assignments are estimated, and paired-input coverage is measured within each group.

Figure 6: Problem-content profile of the retained PhysAlign source problems. The left panel hierarchically groups recurring physics-related vocabulary into broad concept families, while the right panel visualizes prominent terms appearing in the problem text. This lexical view complements the parent-level domain statistics in Table 6 ; it is not used for domain assignment, probe construction, eligibility filtering, or benchmark scoring.

(a) Split-level density and paired-probe coverage
Split	Localized probes / parent	Paired-probe coverage (%) a
Development	2.63	43.27
Test	3.45	14.78
Full release	3.39	16.55

Appendix

Table 7: Probe density, paired-input coverage, and supplementary statistics of PhysAlign. Panel (a) reports split-level density and paired-probe coverage; panel (b) provides full-release summaries with explicit counting units. Paired +GT variants are not additional unique probes.

Model	Reasoning setting	Sampling / decoding	Output cap	Image
Main probe evaluation
Qwen3.5-4B	Native thinking	Greedy	8,192	IMG02
Qwen3.5-9B	Native thinking	Greedy	8,192	IMG02
Qwen3.5-27B	Native thinking	Greedy	8,192	IMG02
InternVL3.5-8B-HF	Explicit CoT	Greedy	8,192	IMG01
Gemini 3.8 Flash	Provider default	$T=0$	8,192	IMG03

Appendix

Table 8: Inference configurations and image-processing policies.

Figure 7: Reusable prompt templates. (a) Local probes share one system instruction and user template; only the task instruction and JSON schema differ across T02-single, T03-text, and T03-image. Angle-bracketed expressions denote instance-specific placeholders.

Figure 8: Paired inputs and original-problem scoring. (b) Base and +GT inputs differ only in the local-reading field. (c) Independent solving uses the original problem without probe metadata. Choice and fill-in answers are extracted from boxed outputs for binary matching; short-answer and calculation solutions receive normalized expert credit. Panel (c) is a scoring schematic, not a verbatim prompt. Angle-bracketed expressions are placeholders.

Model	$\mathrm{GAcc}_{\mathcal{G}}\,\uparrow$	$\mathrm{CAcc}\,\uparrow$	$\mathrm{JAcc}\,\uparrow$	$\mathrm{GAcc}_{\mathcal{L}}\,\uparrow$	$\mathrm{GAcc}\mid C\,\uparrow$	$\mathrm{GErr}\mid C\,\downarrow$
Qwen3.5-4B	40.21	53.97	32.09	46.51	59.46	40.54
Qwen3.5-9B	40.52	65.81	37.55	54.55	57.06	42.94
Qwen3.5-27B	84.11	84.14	68.12	80.29	80.97	19.03
InternVL3.5-8B-HF	23.52	2.12	1.05	33.36	49.37	50.63
GPT-6 Astra	81.94	89.44	77.13	83.31	86.23	13.77
Gemini-3.8-flash	90.71	85.20	70.59	79.33	82.85	17.15

Appendix

Table 9: Full main results (%):

\mathcal{G}

contains 986 parents/3,341 probes;

\mathcal{L}

contains 311 parents/412 probes.

Model	Reading correct, grounding correct $\uparrow$	Reading correct, grounding wrong	Reading wrong, grounding correct	Reading wrong, grounding wrong $\downarrow$
Qwen3.5-4B	32.09	21.88	14.42	31.62
Qwen3.5-9B	37.55	28.26	17.00	17.19
Qwen3.5-27B	68.12	16.01	12.17	3.70
InternVL3.5-8B-HF	1.05	1.07	32.32	65.57
GPT-6 Astra	77.13	12.32	6.19	4.37
Gemini-3.8-flash	70.59	14.61	8.74	6.05

Appendix

Table 10: Joint reading-grounding decomposition on

\mathcal{L}

(311 parents, 412 probes; %).

Model	Base (Raw) $\uparrow$	+GT $\uparrow$	$\Delta$ (pp) $\uparrow$
Qwen3.5-4B	46.51	54.61	+8.10
Qwen3.5-9B	54.55	56.53	+1.98
Qwen3.5-27B	80.29	82.00	+1.71
InternVL3.5-8B-HF	33.36	34.81	+1.45
GPT-6 Astra	83.59	84.90	+1.32
Gemini-3.8-flash	78.67	80.86	+2.20

Appendix

Table 11: Paired Base/+GT grounding on

\mathcal{P}_{m}

(%). Observed parent/probe counts are 311/412 for open-weight models and 304/397 for API models (Appendix B.5 ).

Model	$N/n$	Repaired share	Harmed share	Repair rate	Harm rate	$\Delta G$ (pp)
Qwen3.5-4B	311/412	13.35 $[10.01,17.05]$	5.25 $[3.15,7.68]$	24.96 $[18.92,31.59]$	11.29 $[6.96,16.43]$	+8.10 $[3.88,12.50]$
Qwen3.5-9B	311/412	8.30 $[5.50,11.24]$	6.32 $[3.71,9.10]$	18.26 $[12.44,24.43]$	11.59 $[6.85,16.73]$	+1.98 $[-1.96,5.82]$
Qwen3.5-27B	311/412	5.20 $[3.03,7.43]$	3.48 $[1.58,5.66]$	26.37 $[16.46,36.25]$	4.34 $[1.94,7.02]$	+1.71 $[-1.51,4.74]$
InternVL3.5-8B	311/412	11.17 $[7.83,14.91]$	9.73 $[6.75,13.07]$	16.77 $[11.99,22.06]$	29.16 $[20.87,37.32]$	+1.45 $[-3.73,6.47]$

Appendix

Table 12: Item-level grounding transitions from Base to +GT on the observed paired support of the four open-weight models. Brackets give 95% source-parent-cluster bootstrap intervals.

Baseline	GAcc $\uparrow$	Scope and interpretation
Uniform random candidate	18.69	Chance level under the official aggregation weights
Nearest-region heuristic	76.82	Restricted to probes with a valid spatial distance

Appendix

Table 13: Grounding accuracy of rule-based baselines (%).

Model	Random $GAcc\,\uparrow$	Nearest $GAcc\,\uparrow$	Model $GAcc\,\uparrow$	Model–Nearest (pp)
Qwen3.5-4B	19.84	76.82	49.04	$-27.79$ $[-32.16,-23.43]$
Qwen3.5-9B	19.84	76.82	51.78	$-25.04$ $[-29.12,-20.81]$
Qwen3.5-27B	19.84	76.82	86.35	+9.53 $[6.10,13.06]$
InternVL3.5-8B	19.84	76.82	30.30	$-46.52$ $[-50.58,-42.20]$

Appendix

Table 14: Same-support spatial comparison on

\mathcal{H}

(665 parents/1,639 probes for every row). Accuracies are percentages; model-minus-nearest differences are percentage points with 95% intervals.

(a) Grounding on $\mathcal{H}_{\mathrm{hard}}$ : 271 parents/394 probes
Model	Random $GAcc\,\uparrow$	Model $GAcc\,\uparrow$	Model–Random (pp)
Qwen3.5-4B	18.76	38.03	+19.27 $[12.38,25.89]$
Qwen3.5-9B	18.76	35.21	+16.46 $[9.86,23.09]$
Qwen3.5-27B	18.76	73.22	+54.47 $[48.05,60.45]$
InternVL3.5-8B	18.76	17.80	$-0.96$ $[-6.28,4.56]$

Appendix

Table 15: Diagnostics on the nearest-region failure subset. All shares and accuracies are percentages; differences are percentage points. Brackets give 95% source-parent-cluster bootstrap intervals.

Model	Reference $\uparrow$	No image $\uparrow$	$\Delta_{\mathrm{img}}$ (pp)	Reordered $\uparrow$	$\Delta_{\mathrm{ord}}$ (pp)
Qwen3.5-4B	40.21	38.78	$-1.43$	41.93	$+1.72$
Qwen3.5-9B	40.52	43.53	$+3.01$	43.02	$+2.50$
Qwen3.5-27B	84.11	63.72	$-20.39$	84.48	$+0.37$
InternVL3.5-8B-HF	23.52	49.87	$+26.35$	7.24	$-16.28$
GPT-6 Astra	81.94	75.57	$-6.37$	84.27	$+2.33$
Gemini-3.8-flash	90.71	76.90	$-13.81$	90.82	$+0.11$

Appendix

Table 16: Grounding under input controls (%);

\Delta

denotes percentage-point change from the reference condition.

Model	$\bar{G}_{\mathrm{wrong}}$	$\bar{G}_{\mathrm{correct}}$	$\Delta_{\mathrm{solve}}$ (pp)
Qwen3.5-4B	39.19	41.31	$+2.12$
Qwen3.5-9B	40.67	44.61	$+3.94$
Qwen3.5-27B	85.51	87.89	$+2.39$
InternVL3.5-8B	23.37	25.88	$+2.51$
GPT-6 Astra	83.05	85.53	$+2.48$
Gemini-3.8-flash	92.08	92.36	$+0.28$

Appendix

Table 17: Parent-level grounding by binary final-answer correctness (311 parents, 412 probes; %), distinct from partial-credit

SolveAcc

\Delta_{\mathrm{solve}}

uses unrounded group means and is reported in percentage points.

Figure 9: Case 1: A role mismatch persists despite correct reading. All three Qwen3.5 models correctly read $D$ at anchor R5, yet assign it to the point charge E1 rather than the conducting pipe E2 under both Base and +GT.

Figure 10: Case 2: A grounding error is repaired under +GT. Under Base, InternVL3.5-8B [ Wang et al., 2025b ] misreads the vertical-axis label and selects E1; after receiving the correct reading, it changes its owner prediction to E2.

Figure 11: Case 3: Local grounding and original-problem solving are not equivalent measures. Panels (a) and (c) show a correct solution with incorrect grounding and correct grounding with an incorrect solution, respectively; panel (d) shows a consistent success.

Figure 12: Case 4: A recorded role mismatch has a concrete equation-level consequence under controlled substitution. Qwen3.5-9B [ Qwen Team, 2026 ] reads $M$ correctly but assigns it to the descending cylinder E2 rather than the wedge E1 under both Base and +GT.

Benchmark	General evaluation		Localized diagnosis				Utility
	Input Ctrl.	Interm. Eval.	Fixed Anchor	Rec./Grd. Split	Paired Rec. Ctrl.	Det. Local Score	Solve Link
SeePhys [ Xiang et al., 2025 ]	✓	✗	✗	✗	✗	✗	✗
SeePhys Pro [ Xiang et al., 2026 ]	✓	$\triangle$	✗	✗	✗	✗	✗
PhysicsArena [ Dai et al., 2025 ]	✗	✓	✗	✗	✗	✗	✓
MathVerse [ Zhang et al., 2024 ]	✓	✓	✗	✗	✗	✗	✗
PhysAlign (Ours)	✓	✓	✓	✓	✓	✓	✓

Appendix

Table 18: Protocol-level comparison with closely related benchmarks.

Explore similar work

Multi-Agent Debate and Visual Information Extraction for SeePhys Pro: A 1st-Place Technical Report from ICML 2026 AI4Math Track 3 Challenge
Jul 24, 2026Jiseok Kwak, Suhyeon Jo, Taewoo Kim +3Multi-Agent Debate Cvpr
KAIST
PhyDrawGen: Physically Grounded Diagram Generation from Natural Language
May 28, 2026Nafiul Haque, Syed Nazmus Sakib, Shifat E ArmanDiagrams Visual Generation
Department of Robotics and Mechatronics Engineering, University of Dhaka
PhysFieldBench: Can Multimodal Models Understand Physical Fields?
Sep 28, 2026Yuezhou Ma, Huikun Weng, Jialong Wu +5Multimodal Large Language Models Multimodal Model
School of Software, BNRist, Tsinghua University, China

Jul 24, 2026, cs.LGJ/K move · Enter open · S save

Multi-Agent Debate and Visual Information Extraction for SeePhys Pro: A 1st-Place Technical Report from ICML 2026 AI4Math Track 3 Challenge

Jiseok Kwak, Suhyeon Jo, Taewoo Kim, Yeongmin Kim+2

KAIST

Multi-Agent Debate Cvpr Single-Agent Baselines Modality Gap

Open paper AskPDF arXiv

This technical report presents our approach to Challenge Track~3: SeePhys Pro at the 3rd AI for Math Workshop, where the task is to answer college-level physics questions whose statement and figure may be given partly or entirely as an image. Visual physics problems become substantially harder for large language models when the decisive information resides in a figure rather than in the text, and this modality gap widens as more of the problem migrates into the image. We address the task with a two-stage framework: a visual information extraction stage that re-expresses figure content as solver-readable text to close the modality gap, and a reasoning stage that orchestrates three heterogeneous solvers through multi-agent debate. Our analysis yields two findings: the gain from orchestration comes from reliable answer selection rather than from additional debate, and the value of a figure aid scales with how much of the problem is locked inside the image. The resulting pipeline improves overall accuracy over a single-agent baseline from 0.643 to 0.802 on the public split, and won 1st place on both the public and the private leaderboard (private overall 0.743).

Abstract

Explore similar work

Multi-Agent Debate and Visual Information Extraction for SeePhys Pro: A 1st-Place Technical Report from ICML 2026 AI4Math Track 3 Challenge

PhyDrawGen: Physically Grounded Diagram Generation from Natural Language

PhysFieldBench: Can Multimodal Models Understand Physical Fields?