cs.LGApr 13, 2026
SaveSolving Physics Olympiad via Reinforcement Learning on Physics Simulators
Organizations: Carnegie Mellon University
Abstract
We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question-answer (QA) pairs, a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question-answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.
Figures & tables
Figure 1: Overview of the Sim2Reason (Sim2Reason) pipeline. From left to right: we procedurally generate diverse physics scenes using a DSL, (a) compile them into MuJoCo simulations, and (b) record physically grounded state/force traces. (c–e) From these traces we automatically instantiate multiple types of question–answer pairs (numeric, reverse, and symbolic), and apply filtering to remove degenerate/shortcut questions and unstable simulation segments. (f) Finally, we post-train an LLM with RLVR on the resulting synthetic data and evaluate zero-shot sim-to-real transfer on real-world benchmarks (e.g., IPhO and other physics/math datasets).
Figure 2: Overview of our synthetic data-generation pipeline. We procedurally generate simulatable scenes by randomly selecting and connecting DSL entities (Section 2.1 ), then simulate each scene in MuJoCo and record time-series data of key physical attributes (Section 2.2 ). From these traces we craft natural-language QA pairs in three formats (Section 2.3 )-numeric, reverse, symbolic-and finally deduplicate and filter degenerate/shortcut-solvable questions before RL post-training (Section 2.4 ).
Figure 3: Example of our scene-generation DSL (top) and the corresponding simulator-rendered scene produced by compiling the DSL into simulator scene (bottom). The DSL composes scenes from reusable entities and bodies with explicit connection modes, enabling scalable procedural generation while restricting randomization to physically meaningful parameters.
Figure 4: Illustration of a shortcut solution . The correct answer depends on the coupled motion of the block and wedge , but weaker models may collapse the dotted region into a single body of mass and still match the numeric answer. We filter QA pairs whose answers are invariant to such approximations.
Figure 5: Validation accuracy (green) versus average response length (blue, in tokens) for Qwen3-30B-Instruct over RL post-training steps. Longer responses are strongly associated with higher validation accuracy, suggesting that post-training encourages more extensive intermediate reasoning.
| Model | Synthetic Numeric | Synthetic Symbolic | HCV | IPhO Mechanics |
|---|---|---|---|---|
| Qwen3-30B | 14.8% | 8.8% | 53.9% | 35.6% |
| + RL (synthetic) | 17.4% ( +2.6% ) | 8.0% ( -0.8% ) | 59.0% ( +5.1% ) | 40.0% ( +4.4% ) |
| Qwen2.5-32B | 8.9% | 5.6% | 50.6% | 19.8% |
| + RL (synthetic) | 21.9% ( +13.0% ) | 10.4% ( +4.8% ) | 53.9% ( +3.3% ) | 25.2% ( +5.4% ) |
| Qwen2.5-14B | 7.0% | 5.6% | 49.3% | 16.07% |
| + RL (synthetic) | 17.0% ( +10.0% ) | 10.4% ( +4.8% ) | 51.7% ( +2.4% ) | 20.45% ( +4.4% ) |
Table 1: Performance of Qwen2.5 family Instruct models before and after RL on synthetic datasets, expressed in percentage. Improvements are shown in parentheses.
| Benchmark | Model | Score |
|---|---|---|
| JEEBench | Qwen2.5 32B | 34.38% |
| + RL (synthetic) | 52.28%( +17.90% ) | |
| PHYSICS | Qwen2.5 32B | 39.42% |
| + RL (synthetic) | 43.09%( +3.67% ) | |
| OlympiadBench | Qwen2.5 32B | 41.41% |
| + RL (synthetic) | 44.53%( +3.12% ) |
Table 2: Mean accuracy of Qwen 2.5 32B Instruct on other real world benchmarks.
| Model (Qwen 32B) | Synthetic | IPhO |
|---|---|---|
| Baseline | 14.0% | 19.8% |
| + SFT | 16.0% ( +2.0% ) | 15.9% ( -3.9% ) |
| + RL (Ours) | 32.0% ( +18.0% ) | 25.2% ( +5.4% ) |
Table 3: Comparison of RL vs. SFT on 32B model performance.
Table 4: Ablations on (a) QA format and (b) Data filtration
| Base Model | Post-Trained Model | IPhO Accuracy (%) |
|---|---|---|
| Qwen2.5-32B | DAPO-32B | 24.7 |
| Qwen2.5-32B | LIMO-32B | 25.5 |
| Qwen3-30B | Prime P1 30B | 38.6 |
| Qwen3-30B | Sim2Reason (Ours) | 40. 0 |
Table 5: Comparison with open-weight post-trained models trained on real-world QA datasets.
| Model (Qwen 3B) | IPhO |
|---|---|
| Baseline | 5.68 |
| + RL DAPO-17K (Real) | 9.98 |
| + RL Mixed: DAPO-17K (Real) + Synthetic | 10.35 |
| + RL Synthetic (Ours) | 13.15 |
Table 6: Comparison with a real-world post-training dataset.
| Category | Qwen 32B | + RL (synthetic) |
|---|---|---|
| High School and Below | 65.5% | 68.3% ( +2.8% ) |
| High School Olympiad | 52.9% | 54.0% ( +1.1% ) |
| Undergraduate | 47.9% | 48.4% ( +0.5% ) |
| Postgraduate | 32.2% | 37.8% ( +5.6% ) |
Table 7: Detailed performance across difficulty levels on the PHYSICS benchmark.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Timestep pruning for simulation traces with unmodelled transitions. Left: MuJoCo scene snapshots at the start and at time 3s. Right: recorded time-series signals; when a sliding-window deviation criterion flags an outlier (e.g., due to contact between block and pulley), we keep only the stable prefix (green) and discard the remainder before generating QA pairs.
Figure 9: Correlation between accuracy on Sim2Reason synthetic questions and IPhO mechanics questions.
| Body | Symbol(s) | Description |
|---|---|---|
| Mass | Point mass / block mass. | |
| Sphere | Sphere radius and mass. | |
| Polygonal prism | Number of sides, circumscribed radius, height, and mass. | |
| Cylinder | Cylinder radius, height, and mass. | |
| Disc | Disc radius and mass. | |
| Bar | Bar width, length, height, and mass. |
Table 8: Bodies used by the DSL and the corresponding randomizable parameters.
| Category | Quantity | Description |
| Mass | displacement | Body displacement / position (in world frame). |
| Mass | com_offset | Vector from body frame origin to center of mass. |
| Mass | velocity (6D) | Linear and angular velocity. |
| Mass | acceleration (6D) | Linear and angular acceleration. |
| Mass | mass | Body mass. |
| Mass | momentum (6D) | Linear and angular momentum. |
Table 9: Physical quantities recorded from MuJoCo for each simulated scene.
Figure 29: Scaling up the DSL using LLMs. In this experiment, we task the LLM to extend our entity vocabulary by inventing new entities to support simulation of this question from F=MA 2024 . We observe that while modern LLMs fail to simulate a target scene by generating raw simulator code (left), they can do so by extending our DSL with novel entities (right).
Figure 30: Scaling up the DSL using LLMs. In this experiment, we task the LLM to extend our entity vocabulary by inventing new entities to support simulation of this question from USA PhO 2019 . In this case, we observe that LLMs are able to simulate a target scene by generating raw simulator code (left), and by extending our DSL with novel entities (right).
Figure 31: Scaling up the DSL using LLMs. In this experiment, we task the LLM to extend our entity vocabulary by inventing new entities to support simulation of this question from JEE Advanced 2019 . We observe that while LLMs fail to simulate a target scene by generating raw simulator code (left), they can do so by extending our DSL with novel entities (right).
Figure 32: We task LLMs to port a subset of our entities from MuJoCo simulator to Omniverse simulator, to evaluate the portability of our DSL across simulators. LLMs could successfully transfer DSL entities across simulators, which could potentially help expand the scope of simulator-based synthetic data generation.
More in this topic
Scientific Reasoning in Language Models
Scientific reasoning in language models studies how AI systems use evidence, domain knowledge, and tools to solve problems and support discovery.
See all 133 papers