Reward Design for Physical Reasoning in Vision-Language Models

Derek Lilienthal; Manisha Mukherjee; Sameera Horawalavithana

cs.AIApr 15, 2026

Reward Design for Physical Reasoning in Vision-Language Models

Authors: Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

Organizations: Pacific Northwest National Laboratory, Richland, WA 99352, USA

Abstract

Physical reasoning over images requires integrating perception, domain knowledge, and multi-step symbolic inference, yet Vision-Language Models (VLMs) fall far short of human performance on physics benchmarks. Reinforcement learning can improve reasoning via reward signals, but it remains unclear whether these signals must come from outside the model or can be derived from the model itself. We compare external rewards, which rely on an outside verifier (rule-based format and answer-correctness checks, or an LLM-judge rubric), with an internal attention-based reward computed from the model's own attention over image regions, requiring no labels or judge. We find that external rewards drive most of the accuracy gains, while preserving more coherent reasoning when rewarded for reasoning tokens. The internal attention-based reward, provides an annotation-free grounding signal that improves spatial-relation reasoning and is complementary to external rewards: adding it to format and accuracy rewards, at no additional annotation cost, improves average accuracy on Molmo~2 while generalizing to out-of-distribution tasks. Our results suggest that internal attention rewards can supplement external supervision and lead to improved physical reasoning.

Figures & tables

Figure 1: External and internal rewards for GRPO-based VLM training on physical reasoning. Given a PhyX image-question pair, the model is trained with GRPO using external rewards (verifiable format and accuracy checks, and a rubric-based reward for correctness, principle, and units) or an internal reward computed from its own attention over the image, which needs no labels or judge.

		PhyX							MMMU						SciBench
Method	Avg	P-Ovr	P-EM	P-Mech	P-Mod	P-Opt	P-Th	P-W/A	M-Ovr	M-CM	M-EM	M-NP	M-Opt	M-TSM	S-Fund
Gemma-4-E2B-it
Baseline	0.066	0.047	0.077	0.073	0.049	0.023	0.023	0.036	0.110	0.104	0.096	0.167	0.187	0.057	0.060
SFT	0.057	0.049	0.069	0.062	0.045	0.036	0.041	0.040	0.072	0.069	0.073	0.130	0.053	0.076	0.044
GRPO ( $r_{f}$ )	0.101	0.071	0.067	0.103	0.065	0.046	0.060	0.085	0.152	0.143	0.136	0.167	0.200	0.181	0.082
GRPO ( $r_{f}$ + ASM)	0.080	0.053	0.048	0.079	0.049	0.038	0.037	0.067	0.127	0.119	0.108	0.185	0.167	0.162	0.060

Table 1: Mean judge score on PhyX (P), MMMU (M) and SciBench (S) for baseline and SFT models; scores averaged over 3 runs of each config. Within each model, the best score per column is in bold and the second best underlined. GRPO rows are labeled by their reward terms:

r_{f}

= rule-based format reward,

r_{a}

= LLM-judge answer accuracy,

r_{\text{rub}}

= LLM-judge (rubric) answer and physics reasoning accuracy, ASM = attention-mask score. P- = PhyX: Overall (Ovr), Electromagnetism (EM), Mechanics (Mech), Modern Physics (Mod), Optics (Opt), Thermodynamics (Th), Waves and Acoustics (W/A). M- = MMMU: Overall (Ovr), Classical Mechanics (CM), Electromagnetism (EM), Nuclear Physics (NP), Optics (Opt), Thermodynamics and Statistical Mechanics (TSM). S- = SciBench: Fundamentals of Physics (Fund). Avg = weighted average of the datasets’ overall scores.

Molmo2-O-7B
Method	Overall.	Phys.	Multi.	Spat.	Num.	Pred.	Impl.	Ent. $\downarrow$
Baseline	0.057	0.055	0.030	0.078	0.104	0.039	0.086	0.740
SFT	0.056	0.052	0.041	0.073	0.125	0.015	0.052	0.736
$r_{f}+r_{a}$	0.085	0.082	0.058	0.108	0.068	0.137	0.069	0.773
$r_{f}+r_{a}$ + ASM	0.097	0.093	0.062	0.135	0.083	0.108	0.109	0.683
$\Delta$	+0.041	+0.039	+0.033	+0.057	-0.021	+0.069	+0.023	-0.057

Table 2: PhyX accuracy (judge score) by reasoning type for open-ended questions (top-3 run average), and normalized attention entropy (Ent.; lower = more concentrated) over all PhyX, MMMU, and SciBench test questions for the best checkpoint of each configuration. Reasoning types: Physical Model Grounding (Phys.), Multi-Formula (Multi.), Spatial Relation (Spat.), Numerical (Num.), Predictive (Pred.), and Implicit Condition (Impl.).

\Delta

= last row minus baseline. Best per column and model in bold.

Unperturbed
Reward	Avg	PhyX	MMMU	SciBench
$r_{f}+r_{a}$ + ASM	0.115	0.097	0.155	0.142
Attention noise ( $\sigma$ )
$\sigma=0.1$	0.066	0.050	0.101	0.060
$\sigma=0.5$	0.055	0.039	0.086	0.055
$\sigma=1.0$	0.048	0.034	0.083	0.044

Table 3: Controls for ASM on Molmo 2. All rows use

r_{f}+r_{a}

+ ASM and differ only in the ASM term’s attention map or mask. Accuracy (judge score) is the mean over 3 runs. Noise: Gaussian noise with std

\sigma

added to the normalized attention map. Random mask: a random mask replaces the foreground mask, with the same area (area-matched) or a fixed fraction of patches.

Panel of Figure 2 — Figure 2: Rubric reward analysis for Gemma 4 (Molmo 2 in Appendix Figure 4 ). (a) Mean $\pm$ std over seeds (7-step moving average) against the percentage of training. Left: component scores of format-passing completions, normalized to $[0,1]$ . Right: rubric correctness (divided by 2) over all completions. (b) Share of the within-group reward variance attributable to each rubric term in the early, middle, and late thirds of training.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

		PhyX							MMMU						SciBench
Method	Avg	P-Ovr	P-EM	P-Mech	P-Mod	P-Opt	P-Th	P-W/A	M-Ovr	M-CM	M-EM	M-NP	M-Opt	M-TSM	S-Fund
Gemma-4-E2B-it
Baseline	0.066 $\pm$ 0.004	0.047 $\pm$ 0.002	0.077 $\pm$ 0.006	0.073 $\pm$ 0.003	0.049 $\pm$ 0.000	0.023 $\pm$ 0.010	0.023 $\pm$ 0.007	0.036 $\pm$ 0.006	0.110 $\pm$ 0.010	0.104 $\pm$ 0.009	0.096 $\pm$ 0.004	0.167 $\pm$ 0.056	0.187 $\pm$ 0.031	0.057 $\pm$ 0.029	0.060 $\pm$ 0.009
SFT	0.057 $\pm$ 0.004	0.049 $\pm$ 0.004	0.069 $\pm$ 0.007	0.062 $\pm$ 0.015	0.045 $\pm$ 0.009	0.036 $\pm$ 0.010	0.041 $\pm$ 0.004	0.040 $\pm$ 0.009	0.072 $\pm$ 0.011	0.069 $\pm$ 0.009	0.073 $\pm$ 0.008	0.130 $\pm$ 0.032	0.053 $\pm$ 0.050	0.076 $\pm$ 0.016	0.044 $\pm$ 0.019
GRPO ( $r_{f}$ )	0.101 $\pm$ 0.045	0.071 $\pm$ 0.041	0.067 $\pm$ 0.033	0.103 $\pm$ 0.060	0.065 $\pm$ 0.049	0.046 $\pm$ 0.026	0.060 $\pm$ 0.046	0.085 $\pm$ 0.034	0.152 $\pm$ 0.044	0.143 $\pm$ 0.034	0.136 $\pm$ 0.055	0.167 $\pm$ 0.000	0.200 $\pm$ 0.072	0.181 $\pm$ 0.072	0.082 $\pm$ 0.043
GRPO ( $r_{f}$ + ASM)	0.080 $\pm$ 0.055	0.053 $\pm$ 0.044	0.048 $\pm$ 0.046	0.079 $\pm$ 0.068	0.049 $\pm$ 0.043	0.038 $\pm$ 0.055	0.037 $\pm$ 0.016	0.067 $\pm$ 0.042	0.127 $\pm$ 0.080	0.119 $\pm$ 0.059	0.108 $\pm$ 0.092	0.185 $\pm$ 0.085	0.167 $\pm$ 0.110	0.162 $\pm$ 0.100	0.060 $\pm$ 0.034

Appendix

Table 4: Mean

\pm

sample standard deviation of the judge score over the top-3 runs of each configuration on PhyX (P), MMMU (M) and SciBench (S); a dash after

\pm

means only one run. GRPO rows are labeled by their reward terms:

r_{f}

= rule-based format reward,

r_{a}

= LLM-judge (jury) answer accuracy,

r_{\text{rub}}

= LLM-judge (rubric) answer and physics reasoning accuracy, ASM = attention-mask score. Column labels are <dataset>-<category> . P- = PhyX: Overall (Ovr), Electromagnetism (EM), Mechanics (Mech), Modern Physics (Mod), Optics (Opt), Thermodynamics (Th), Waves and Acoustics (W/A). M- = MMMU: Overall (Ovr), Classical Mechanics (CM), Electromagnetism (EM), Nuclear Physics (NP), Optics (Opt), Thermodynamics and Statistical Mechanics (TSM). S- = SciBench: Fundamentals of Physics (Fund). Avg = sample-weighted average of the datasets’ overall scores.

Domain	Train	Test	Total
Electromagnetism	382	168	550
Mechanics	382	168	550
Optics	342	158	500
Thermodynamics	339	161	500
Waves/Acoustics	335	165	500
Modern Physics	236	164	400

Appendix

Table 5: PhyX open-ended questions per physics domain in the train and test splits.

Principle (subfield)	Train	Test	Total	Principle (subfield)	Train	Test	Total
Geometrical Optics	270	127	397	Statics	57	21	78
Laws of Thermodynamics	189	90	279	Magnetism	50	25	75
Wave Properties	186	87	273	Rotational Motion	33	39	72
Electrostatics	145	81	226	Sound	54	18	72
Quantum Phenomena	100	101	201	Ideal Gases and Kinetic Theory	33	24	57
Dynamics	154	34	188	Momentum and Collisions	20	9	29

Appendix

Table 6: Principle labels: the PhyX subfield of each question, which the rubric judge receives as a coarse hint about the area of physics (it accepts any principle that correctly governs the problem). Counts of open-ended questions in the train and test splits (27 labels).

Quantity category	Train	Test	Total	Quantity category	Train	Test	Total
Length	441	211	652	Electric Current	40	18	58
Energy	217	106	323	Pressure	26	26	52
Velocity	212	104	316	Mass	33	17	50
Angle	123	82	205	Acceleration	36	11	47
Force	149	48	197	Magnetic Field (Flux Density)	26	18	44
Time	119	51	170	Electric Charge	20	12	32

Appendix

Table 7: Unit labels grouped by physical quantity category (71 categories in total; the 20 most frequent are listed). Counts of open-ended questions in the train and test splits.

Unit	Train	Test	Total	Unit	Train	Test	Total
meter	428	202	630	hertz	79	40	119
dimensionless	152	99	251	volt	66	21	87
meter per second	158	69	227	watt	55	12	67
joule	157	66	223	ampere	40	17	57
newton	147	46	193	electronvolt	38	18	56
second	102	48	150	Other (123 units)	501	291	792

Appendix

Table 8: Most frequent ground-truth unit strings supplied to the rubric judge (135 distinct units; 7 questions have no unit label because the answer is symbolic or qualitative).

Dataset	Split	# images	Whitespace (%)
PhyX	Train	2,016	$80.4\pm 20.3$
PhyX	Test	984	$79.6\pm 20.2$
MMMU Physics	Test	424	$90.3\pm 9.9$
SciBench	Test	177	$76.2\pm 29.6$
Average over datasets		3,601	$81.6$
Average over all images			$81.1\pm 20.2$

Appendix

Table 9: Percentage of image pixels classified as whitespace by the ASM reward (all RGB channels

\geq 230

), mean

\pm

standard deviation over images.

Figure 3: Mean whitespace percentage per dataset. Error bars show the standard deviation over images (clipped at 100%); the last bar is the mean of the four dataset means.

Dataset	Comparison	$n$	Mean $H$ (a)	Mean $H$ (b)	$\Delta H$ [95% CI]	Lower
Molmo2-O-7B
All	$r_{f}+r_{a}$ $\to$ $r_{f}+r_{a}$ + ASM	1584	0.773	0.683	$-0.090$ $[-0.097,-0.083]$	71%
PhyX		984	0.789	0.708	$-0.080$ $[-0.089,-0.072]$	68%
MMMU		423	0.753	0.649	$-0.103$ $[-0.116,-0.090]$	78%
SciBench		177	0.735	0.626	$-0.109$ $[-0.133,-0.085]$	73%
All	Baseline $\to$ $r_{f}+r_{a}$	1584	0.740	0.773	$+0.033$ $[+0.026,+0.041]$	48%

Appendix

Table 10: Paired change in normalized attention entropy

H

\to

b; negative = more concentrated attention) with 95% normal-approximation confidence intervals over questions, for the best checkpoint of each configuration. “Lower” is the share of questions whose entropy decreases.

	Molmo2-O-7B			Gemma-4-E2B-it
Component	$P(c\mid\text{hi})$	$P(c\mid\text{lo})$	$r$	$P(c\mid\text{hi})$	$P(c\mid\text{lo})$	$r$
Principle	0.080	0.009	0.26	0.188	0.014	0.42
Unit	0.077	0.043	0.24	0.183	0.092	0.35
Reasoning	0.103	0.005	0.64	0.233	0.005	0.72

Appendix

Table 11: Predictive validity of the rubric components for correctness among format-passing training completions.

P(c\mid\text{hi})

and

P(c\mid\text{lo})

: probability that a completion is correct (judge correctness

\geq 1

of 2) when the component is high (principle/unit awarded; reasoning

\geq 1

of 2) or low.

r

: within-group Pearson correlation with correctness.

	Molmo2-O-7B		Gemma-4-E2B-it
Reward variant	Signal (%)	Adv. corr.	Signal (%)	Adv. corr.
Correctness only	27.8	0.89	39.4	0.94
Corr. + Principle	58.4	0.94	62.6	0.96
Corr. + Unit	54.4	0.87	57.4	0.91
Corr. + Reasoning	87.1	0.81	94.4	0.87
Principle + Unit + Reasoning	88.2	0.93	93.5	0.92

Appendix

Table 12: Offline reweighting of logged rubric components. For each reward variant we recompute GRPO advantages on the logged groups. Signal: percentage of groups whose reward has non-zero variance; Adv. corr.: correlation between the variant’s and the full rubric’s advantages.

Figure 4: Rubric reward training dynamics for Molmo 2 (Gemma 4 in Figure 2 a). Mean $\pm$ std over seeds (7-step moving average) against the percentage of training. Left: component scores of format-passing completions, normalized to $[0,1]$ . Right: rubric correctness (divided by 2) over all completions.

Figure 5: Share of the within-group reward variance attributable to each rubric term for Molmo 2 (Gemma 4 in Figure 2 b), in the early, middle, and late thirds of training.

Model	ASM overhead (s/step)	ASM overhead (%)
Molmo2-O-7B	$+26.9$	$+10$
Gemma-4-E2B-it	$+33.2$	$+11$

Appendix

Table 13: Length-adjusted wall-clock overhead of the ASM reward per training step (maximum completion length 2048), estimated by regressing the median step time of each open-ended run (

r_{f}

r_{f}+r_{a}

r_{\text{rub}}

, with and without ASM) on its mean completion length, the reward type, and an ASM indicator. The 95% bootstrap intervals over runs include zero for both models.

Model and data
Component	Gemma 4 E2B-it	Molmo2-O (7B)
Initialization	Pretrained instruction-tuned checkpoint, no SFT warmup
Training data	PhyX open-ended, 2,016 training problems
Evaluation	PhyX testmini (984 problems); MMMU physics and SciBench zero-shot
GRPO hyperparameters
Rollout generations	$G=8$ per prompt	$G=8$ per prompt

Appendix

Table 14: Training setup for the GRPO experiments and the SFT baseline.

Benchmark	Responses	Agreement (%)	GPT-oss acc.	Haiku acc.
PhyX	57,072	98.5	0.075	0.079
MMMU	24,592	95.5	0.161	0.140
SciBench	10,266	98.6	0.110	0.115
All	91,930	97.7	0.102	0.099

Appendix

Table 15: Agreement between GPT-oss 120B and Claude Haiku 5.5 on the open-ended evaluation outputs of all re-scored runs, and the mean accuracy under each judge.

Figure 6: Cumulative image attention over the generated tokens for Gemma-4-E2B-it on an MMMU physics test question, for the baseline, SFT, GRPO ( $r_{f}+r_{a}$ ), and GRPO ( $r_{f}+r_{a}$ + ASM). Red indicates lower and green higher attention. ASM is the mean per-token attention-mask score and Avg Entropy is the normalized entropy of the cumulative map.

Figure 7: Cumulative image attention over the generated tokens for Molmo2-O-7B on a PhyX optics question (refraction at a spherical surface), for the baseline, SFT, GRPO ( $r_{f}+r_{a}$ ), and GRPO ( $r_{f}+r_{a}$ + ASM). Red indicates lower and green higher attention. ASM is the mean per-token attention-mask score and Avg Entropy is the normalized entropy of the cumulative map.

Figure 8: Cumulative image attention over the generated tokens for Molmo2-O-7B on a PhyX mechanics question (a disk attached to a spring), for the baseline, SFT, GRPO ( $r_{f}+r_{a}$ ), and GRPO ( $r_{f}+r_{a}$ + ASM). Red indicates lower and green higher attention. ASM is the mean per-token attention-mask score and Avg Entropy is the normalized entropy of the cumulative map.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 841 papers

Aug 22, 2026

Decoupled Physical Modeling and Execution for Physics Reasoning

Physics reasoning requires constructing a consistent model of the underlying physical system rather than relying solely on symbolic or formula-based manipulation. Although large language models have shown strong ability in solving math and coding problems, they still struggle with physics problems, as these problems entangle the physical modeling process with mathematical calculations. Humans approach physics by first building a representation of the system before performing calculations. Inspired by this, we introduce a unified framework that distills intermediate representations that explicitly encode the physical modeling process and adopt a two-stage post-training strategy, where supervised fine-tuning establishes structured modeling, and reinforcement learning with rubric-based feedback improves the quality of the modeling process. Experiments on multiple multimodal physics benchmarks show that our approach generally improves physical reasoning performance across different models and datasets. Across PhysReason, PhyX, and SeePhys, physical modeling outperforms GRPO by ~3% on average. showing that explicit physical modeling is an effective strategy for improving physics reasoning in small VLMs.

Ye Zhang, Xuehang Guo, Rui Pan +4

University of Pennsylvania · William & Mary · University of Illinois Urbana-Champaign +3

Physical Reasoning VLM Reasoning

Ask PDF

Jul 7, 2026

Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment

Vision-language models (VLMs) struggle to generalize in interactive physical reasoning, particularly under unseen tasks and environments. Two key failure modes are prominent: hallucinated chain-of-thought (CoT) reasoning that contradicts physical reality, and misalignment between the model's reasoning and actions. We present VAORA (Visual Action Outcome Reasoning Alignment), a novel reward design that directly addresses both issues. VAORA introduces two complementary rewards: Visual Alignment Reward, which anchors VLM reasoning to the visual context independent of the agent action itself, and Visual-Action Alignment Reward, which grounds reasoning in the visual outcome induced by the model's action. Together, these rewards suppress hallucinated CoT and reduce the gap between reasoning and behavior. To improve training stability, we further employ smooth, dense rewards by estimating success probabilities using a pre-trained in-domain expert agent. Experiments on PHYRE and Virtual Tool support our performances across novel-task and unseen-environment settings, confirming that grounded and generalizable physical intelligence can be induced through VAORA.

Han-Jun Ko, Jr-Jen Chen, Haobo Yuan +4

National Taiwan University · The University of California, Merced

Physical Reasoning VLM Reasoning Visually Grounded Reasoning RL for VLMs Embodied Reasoning

Ask PDF

Oct 15, 2026

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty is that visual task failures can arise from different sources: the same benchmark may be limited by perception for one model and by reasoning for another. We introduce Perception and Reasoning Attribution (PARA), which measures the remaining perceptual and reasoning headroom of each model--benchmark pair by comparing how many current errors are recovered with more accurate visual information versus a stronger reasoner. Across 4 vision-language models and 17 multimodal benchmarks, PARA broadly agrees with the expected characteristics of existing benchmarks while revealing substantial model-specific variation. More importantly, it shows that RL gains become smaller as perceptual headroom increases, but larger when reasoning explains more of the recoverable errors, suggesting that current RL exploits reasoning headroom more effectively than perceptual headroom. Motivated by this gap, we analyze how existing RL objectives affect visual perception and find that reward- and token-level signals influence visual attention only indirectly. We therefore propose GRAD, a group-relative attention distillation objective that uses the best rollout in each group to directly guide the visual attention of weaker rollouts. GRAD consistently improves performance across two model scales and seven benchmarks, and further complements existing RL methods.

Zeyuan Yang, Zhuohao Chen, Jiaben Chen +3

UMass Amherst · UIUC · MIT

Attention-Guided Distillation RL for VLMs Vision-Language Models VLM Reasoning Reasoning for VLMs VLM Evaluation+1

Ask PDF

Abstract

VLM Reasoning

Explore similar work

Decoupled Physical Modeling and Execution for Physics Reasoning

Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs