Authors: Xinye Yang, Zhenyang Liu, Yuxuan Wang, Yuanyuan Lei
Organizations: Northwestern University, Evanston, IL · Department of Physics, University of Florida, Gainesville, FL · Computer & Information Science and Engineering, University of Florida, Gainesville, FL
Recent research-level physics-reasoning benchmarks for LLMs reveal a failure mode that lies not in knowledge or execution, but in modeling the world being reasoned about. Models often silently alter given quantities, fabricate laws, or inject unstated assumptions to force a problem into familiar patterns, and then reason inside this corrupted world. We define this premise-level divergence as modeling drift: the gap between a model's internal axiomatic world model (AWM) and the true physical world specified in the problem. Although RLVR and PRM have yielded substantial gains in mathematics and code, their reward shapes do not directly supervise world modeling in axiomatic domains such as physics. We propose Reinforcement Learning with Axiomatic World Modeling (RLAWM), which optimizes the policy by penalizing divergence between its inferred axiomatic world model and the true physics world. RLAWM probes physics reasoning at two hierarchical levels (modeling and solving), each scored by a physically grounded reward. The modeling reward targets abstract knowledge formulation, evaluating whether the model captures the true axiomatic world before attempting calculation. The solving reward targets instantiated knowledge reasoning, checking whether the model's concrete derivation remains consistent with the axiomatic world. This two-level structure separates what world the model believes it is reasoning in from how it reasons within that world. The framework operates exclusively during training, with no extra cost during inference. RLAWM improves performance by up to +12.2 points on PhysReason and +5.2 on PHYSICS over the baseline. It also generalizes zero-shot across difficulty, source, and domain, with a 2.45x gain and +10 points over the strongest baseline on olympiad-level physics and math.
Figures & tables
Figure 1: Modeling drift on a representative physics problem. Left: the true axiomatic world model (AWM): objects, geometry, field, and laws. Middle: a representative drift trajectory of the base model: wrong geometry propagates into a malformed energy equation, and upon deriving vO2<0 the model silently sets E←0 for self-consistency instead of repairing the geometry. Right: an RLAWM-trained model recovers the correct geometry and reasons consistently within the true AWM.
Figure 2: The RLAWM pipeline. RLAWM probes physical reasoning at two hierarchical levels, each carrying a grounded reward. Modeling (top) concerns how the policy builds its world. It externalizes the AWM as a five-field [modeling] block, which RM scores against the problem’s ground-truth AWM. Solving (bottom) concerns how the policy executes reasoning over that AWM: it writes its reasoning as pseudocode, where a frozen checker probes for physical invariants, trace consistency, and response under perturbations to yield RS .
PhysReason
PHYSICS-text_only
Method
Easy
Medium
Difficult
Knowledge
Overall
Stat. Mech.
Quantum
Atomic
E&M
Mechanics
Optics
Overall
Backbone: Qwen3-VL-8B-Instruct
Vanilla LLM
39.5
15.5
13.3
46.0
30.0
16.3
17.3
9.7
33.8
18.0
16.7
18.5
RLVR
39.1
22.5
17.5
47.3
32.8
16.8
16.3
14.0
25.0
16.0
36.1
19.2
Direct-Likert
43.8
25.7
20.6
52.8
37.1
23.7
23.8
14.4
33.8
19.7
31.6
23.8
Reference-Likert
31.0
20.1
20.3
55.6
33.2
18.8
22.3
14.4
31.6
18.2
34.2
22.2
Table 1: SubPass@1 (%) on PhysReason and PHYSICS-text_only . Best per column within each backbone is in bold . RLAWM’s RM with gpt-4o-mini-2024-07-18 as the scorer.
Figure 3: SubPass@ k curves across k∈{1,5,10,20,50,100} on PhysReason. As k grows, RLVR and Reference-Likert fall below the base model. RLAWM stays on top at every k .
PhysReason
PHYSICS-text_only
Reward
Easy
Medium
Difficult
Knowledge
Overall
Stat. Mech.
Quantum
Atomic
E&M
Mechanics
Optics
Overall
Backbone: Qwen3-VL-8B-Instruct
RM+RA
48.3
39.8
17.6
63.0
44.0
28.2
19.7
18.2
33.5
20.0
36.1
25.1
RS+RA
47.6
36.3
24.2
66.0
45.3
25.4
19.4
19.5
37.3
17.2
44.4
25.7
Joint RM+RS+RA †
57.6
39.6
23.9
64.0
47.9
26.4
21.9
28.0
42.7
20.5
38.9
29.0
Backbone: Qwen3-VL-4B-Instruct
Table 2: Per-stage reward ablation. SubPass@1 (%); best per column per backbone in bold .
Table 6
Qwen3-VL-8B-Instruct
Qwen3-VL-4B-Instruct
RM
RM ’s Scorer
Easy
Medium
Difficult
Knowledge
Overall
Easy
Medium
Difficult
Knowledge
Overall
PMF
4o
57.6
39.6
23.9
64.0
47.9
51.5
37.0
23.1
61.3
44.8
PMF
qwen
55.1
31.6
20.5
68.0
45.8
44.8
36.6
17.4
65.3
43.0
Judge
4o
57.3
36.5
21.3
66.0
47.1
50.5
34.2
19.7
64.0
44.0
Judge
qwen
51.4
30.9
19.8
66.7
44.2
43.8
31.1
16.1
62.0
40.2
Table 5: Modeling-scorer ablation (PhysReason). SubPass@1 (%); best per column within each backbone in bold . 4o : gpt-4o-mini ; qwen : Qwen3-VL-4B .
Reward
Setting
Easy
Medium
Difficult
Knowledge
Overall
Reference-Likert
Zero-RL
41.7
17.9
10.9
48.7
31.4
Reference-Likert
M/S SFT
43.8
18.6
11.1
51.3
32.9
RLAWM
Zero-RL
49.2
23.9
15.8
54.0
37.3
Table 6: Reward signal vs. SFT data (PhysReason, Qwen3-VL-8B). SubPass@1 (%). Zero-RL : GRPO without SFT. M/S SFT : Reference-Likert trained on RLAWM’s restructured SFT data.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Symbol
Description
Modeling reward (§ 3.2 )
Policy
πθ
LLM-parameterized policy
Externalized AWM
m^
Policy’s structured externalization of its belief (the five-field [modeling] block)
Reference structure
m∗
An operational reference representation constructed from the problem statement and its ground-truth solution. It serves as a proxy for the problem’s physical axiom set rather than as a separately formalized axiomatic theory.
Per-motif F1
F1QEL/VCL/AEB
PMF motif scores
Modeling reward
RM
RMPMF (PMF F1 over the externalized AWM)
Appendix
Table 7: Summary of core notation used throughout the paper.
text-only, English, physics, with verifiable ground truth ( OE_TO_physics_en_COMP )
PHYBench [ 21 ]
MIT
test
500
full set; original text-only problems written by physics students, symbolic-expression answers, high-school to olympiad difficulty
Appendix
Table 8: Dataset overview. We split PhysReason and PHYSICS-text_only randomly 9:1 into training and test partitions; OlympiadBench-Phys, PHYBench, and AIME 2026 are evaluation-only. PhysReason categories combine three reasoning-difficulty tiers ( easy / medium / difficult ) with one knowledge-recall category ( knowledge ); PHYSICS-text_only subdomains are taken directly from the source dataset and listed in test-split frequency order. PhysReason, PHYSICS, OlympiadBench, and PHYBench are released under the MIT License. AIME 2026 is taken from MathArena and released under CC BY-NC-SA 4.0.
Model
Role
License
Qwen3-VL-8B-Instruct [ 1 ]
Policy backbone (main and ablations)
Apache 2.0
Qwen3-VL-4B-Instruct [ 1 ]
Policy backbone; compact PMF scorer (ablation)
Apache 2.0
Gemma-3-12B-IT [ 24 ]
Policy backbone (cross-family verification)
Gemma Terms of Use
Qwen2.5-Coder-1.5B-Instruct [ 9 ]
Frozen translator f (pseudocode → Python)
Apache 2.0
gpt-4o-mini-2024-07-18 [ 18 ]
PMF scorer for RM ; Judge baseline
Proprietary (OpenAI API ToS)
Appendix
Table 9: Pre-trained model assets used in this paper.
Hyperparameter
Value
Fine-tuning type
Full fine-tuning
Vision tower
Frozen
Multimodal projector
Trainable
Language model
Trainable
Cutoff length
8192
Maximum samples
5000
Appendix
Table 10: Main SFT hyperparameters shared by all methods.
Hyperparameter
Value
Advantage estimator
GRPO
Normalize GRPO advantages by std.
Yes
KL in reward
No
Actor KL loss
Enabled
KL coefficient
0.001
KL loss type
Low-variance KL
Appendix
Table 11: Main GRPO hyperparameters shared by RLAWM and all RL baselines.
Run
Overall
Direct-Likert (strongest baseline)
37.1
RLAWM, seed 10
49.1
RLAWM, seed 42 (reported)
47.9
RLAWM, seed 100
47.3
Mean ± std
48.1±0.9
Appendix
Table 12: Seed robustness of RLAWM on PhysReason. SubPass@1 Overall (%) on Qwen3-VL-8B-Instruct; seed 42 is the run reported in Table 1 .
Error type
QEL
VCL
AEB
Wrong theorem selected
Decreases
May decrease
Unchanged
Correct theorem but missing variable
Decreases
May decrease
Unchanged
Broken interaction between equations
Unchanged
Decreases
Unchanged
Missed premise or assumption
Unchanged
Unchanged
Decreases
Missed object or physical stage
Decreases
Decreases
May decrease
Appendix
Table 13: Diagnostic effect of common modeling errors on the three PMF motif components. “May decrease” means that the component is affected only when the local error changes the corresponding motif type.
Failure type
Share of all rollouts
Sandbox execution error
11.8%
Variable-contract mismatch
7.0%
Pseudocode translation failure
0.4%
Missing solving code
0.2%
Total (scored RS=0 )
19.4%
Appendix
Table 14: Failure types behind RS=0 . Share of all training rollouts on PhysReason with Qwen3-VL-8B-Instruct.
Physics reasoning requires constructing a consistent model of the underlying physical system rather than relying solely on symbolic or formula-based manipulation. Although large language models have shown strong ability in solving math and coding problems, they still struggle with physics problems, as these problems entangle the physical modeling process with mathematical calculations. Humans approach physics by first building a representation of the system before performing calculations. Inspired by this, we introduce a unified framework that distills intermediate representations that explicitly encode the physical modeling process and adopt a two-stage post-training strategy, where supervised fine-tuning establishes structured modeling, and reinforcement learning with rubric-based feedback improves the quality of the modeling process. Experiments on multiple multimodal physics benchmarks show that our approach generally improves physical reasoning performance across different models and datasets. Across PhysReason, PhyX, and SeePhys, physical modeling outperforms GRPO by ~3% on average. showing that explicit physical modeling is an effective strategy for improving physics reasoning in small VLMs.
Ye Zhang, Xuehang Guo, Rui Pan +4
University of Pennsylvania · William & Mary · University of Illinois Urbana-Champaign +3
We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question-answer (QA) pairs, a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.
Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.