BeamPERL: Parameter-Efficient RL with Verifiable Rewards Specializes Compact LLMs for Structured Beam Mechanics Reasoning
Authors: Tarjei Paule Hage, Markus J. Buehler
Organizations: Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Civil and Environmental Engineering, Department of Mechanical Engineering, Schwarzman College of Computing, Massachusetts Institute of Technology, Cambridge, MA, USA
Can reinforcement learning with hard, verifiable rewards teach a compact language model to reason about physics, or does it primarily learn to pattern-match toward correct answers? We study this question by training a 1.5B-parameter reasoning model on beam statics, a classic engineering problem, using parameter-efficient RLVR with binary correctness rewards from symbolic solvers, without teacher-generated reasoning traces. The best BeamPERL checkpoint achieves a 66.7% improvement in Pass@1 over the base model. However, the learned competence is anisotropic: the model generalizes compositionally (more loads) but fails under topological shifts (moved supports) that require the same equilibrium equations. Intermediate checkpoints yield the strongest reasoning, while continued optimization degrades robustness while maintaining reward. These findings reveal a key limitation of outcome-level alignment: reinforcement learning with exact physics rewards induces procedural solution templates rather than internalization of governing equations. The precision of the reward signal - even when analytically exact - does not by itself guarantee transferable physical reasoning. Our results suggest that verifiable rewards may need to be paired with structured reasoning scaffolding to move beyond template matching toward robust scientific reasoning.
Figures & tables
Figure 1: Schematic of a simply supported beam defined on the one-dimensional domain x∈[0,L] , where x denotes the axial coordinate along the beam and L is the beam length. The beam is supported by a pinned support located at xpin∈[0,L] and a roller support located at xroller∈[0,L] , with xpin=xroller . The beam is subjected to N transverse point loads Pi , applied at positions xi∈[0,L] , with xi=xj∀i=j , for i=1,…,N , where in this schematic N=2 .
Figure 2: End-to-end dataset generation pipeline for beam-mechanics problem-solving. Discrete beam configurations are first sampled from a symbolic parameter space and solved analytically using a symbolic mechanics solver to obtain exact ground-truth support reactions. An LLM is used to generate multiple natural-language problem formulations for each beam configuration, while the underlying physical solution remains unchanged. This many-to-one construction yields a verifiable question–answer dataset with multiple linguistic variants mapped to a single mechanically correct solution.
Figure 3: PE-RLVR-FT workflow for adapting a distilled LRM to beam-mechanics problem-solving. Beam-mechanics questions from the synthetic dataset are used to prompt a frozen base model augmented with trainable LoRA adapters. For each prompt, the model samples a group of G candidate responses, which are evaluated by a deterministic reward function based on format adherence and symbolic beam statics correctness, each assigning a binary reward. These rewards are converted into relative advantage signals A1,…,AG and used by GRPO to update only the LoRA parameters, while all pretrained backbone weights remain frozen. A more detailed description of this process is provided in Section 4.3 . Adapted from [ 14 ] .
Figure 4: Training performance against the cumulative number of training examples. (a) Total weighted reward. (b) Unscaled format and accuracy rewards. All rewards are averaged across all rollouts for all observed examples at each training step. Both sub-rewards increase sharply during the early training phase, after which the accuracy reward slightly decreases while the format reward remains consistently high, indicating that early performance gains are primarily associated with improved output formatting.
Figure 5: Evolution of training reward, completion length, and KL divergence plotted against the cumulative number of training examples. In (a), reward and rollout completion length are shown on separate y-axes, with completion length, in tokens, decreasing early in training and stabilizing at a task-appropriate token length. In (b), reward and KL divergence are shown on separate y-axes, with KL divergence increasing conservatively at first and more sharply at later stages. All values are averaged across all rollouts for all observed examples at each training step.
Model
Pass@1
Pass@7
Maj@7
DeepSeek-R1-Distill-Qwen-1.5B
12.50%
29.17%
0.00%
BeamPERL (best-performing checkpoint)
20.83%
41.67%
4.17%
Table 1: Performance comparison between the base model and BeamPERL (best-performing checkpoint) across the evaluation metrics. Results are reported in terms of Pass@1, Pass@7, and Majority@7 accuracy, allowing for one-shot, best-case, and output consistency performance analysis.
Figure 6: Evaluation performance across ten equally spaced training checkpoints. Each metric is averaged across all ID and OOD examples. Pass@1 and Pass@7 show that both one-shot and best-case performance initially increase before a slight subsequent decline, while Majority@7 shows a steady, gradual improvement throughout training.
Figure 7: Evaluation performance partitioned across three plots: (a) ID samples consisting of beams supported at the ends with a single load, (b) OOD samples with multiple loads applied at the ends, and (c) OOD samples with varying support locations. Each metric is averaged over the corresponding evaluation examples. Results show improved one-shot performance and increased output consistency on ID samples as training progresses, while generalization across OOD dimensions differs: performance increases for increasing load multiplicity throughout training but degrades for varying support locations when training extends beyond the optimal regime.
Model
AMC23
AIME24
AIME25
DeepSeek-R1-Distill-Qwen-1.5B
72.5%
33.3%
23.3%
BeamPERL (best-performing checkpoint)
75.0%
40.0%
23.3%
Table 2: Performance comparison between the base model and BeamPERL’s best-performing checkpoint on standard mathematical reasoning benchmarks AMC23, AIME24, and AIME25 to assess the effect of task-specific fine-tuning on general reasoning performance. Results indicate that intermediate-stage fine-tuning does not degrade general mathematical reasoning.
Figure 8: Performance on standard mathematical reasoning benchmarks across selected training checkpoints, including initialization, the best-performing BeamPERL checkpoint, a mid-training checkpoint, and the final checkpoint. Accuracy is reported on AMC23, AIME24, and AIME25 to assess the effect of task specialization on general reasoning ability. Results indicate that early-stage training primarily enhances task-specific behavior without disrupting general reasoning performance, while a noticeable decline in benchmark accuracy emerges at later stages of training.
Figure 9: Mean reward trajectories plotted against the cumulative number of training examples for ID and two OOD evaluation groups, beams with multiple applied loads and beams with varying support locations. Panel (a) shows the evolution of the mean format reward, while panel (b) shows the evolution of the mean accuracy reward. ID examples exhibit rapid convergence to near-perfect format adherence in panel (a), followed by a delayed but steady increase in accuracy in panel (b). In contrast, both OOD groups show an initial joint improvement in format and accuracy, after which format adherence declines once saturation is reached. This decline is accompanied by a gradual reduction in accuracy for OOD examples with varying support locations, whereas accuracy for OOD examples with multiple applied loads continues to improve, highlighting distinct generalization behaviors across OOD dimensions.
Figure 10: Relationship between training dynamics and evaluation performance during fine-tuning. (a) Pass@7 evaluation performance plotted against the training reward, illustrating a strong coupling between reward optimization and downstream task performance. (b) Pass@7 performance plotted against the KL divergence from the base model distribution, with the KL term scaled by one half. Early in training, performance gains coincide with low KL divergence, indicating improvements achieved with minimal deviation from the pretrained model. Beyond the best-performing checkpoint, KL divergence increases sharply and becomes highly variable, signaling pronounced distributional shifts associated with later-stage specialization.
Figure 11: Evolution of mathematical benchmark performance under PE-RLFT. (a) Benchmark performance for eight models reported in the Tina project plotted against normalized training progress, with each checkpoint mapped to a common [0,1] scale based on the total number of training steps in the corresponding run. Individual checkpoints are shown together with the envelope defined by the minimum and maximum performance for each normalized checkpoint and the mean trajectory across all models. (b) Comparison between the mean benchmark performance trajectory of the Tina models and the corresponding trajectory obtained in the present BeamPERL study, with training progress normalized in both cases.
Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.
Reinforcement learning is critical to improving large reasoning models, but its success relies heavily on verifiable rewards (RLVR), making it hard to use in open-ended domains where correctness is ambiguous and cannot be verified. Moreover, reasoning trajectories remain largely unconstrained, and optimizing solely toward the final answer can favor early exploitation over generalization. In this work, we ask whether general reasoning ability can be improved by teaching models how to think (the structure of reasoning) rather than what to produce (the outcome of reasoning), and we extend traditional RLVR to open-ended settings. We introduce Structure-Aware Reinforcement Learning (SARL), a label-free framework that constructs per-response reasoning maps from intermediate thinking steps and rewards their reasoning topology. SARL shifts supervision from destination to path, encouraging reasoning trajectories that are both locally coherent and globally efficient. On verifiable math tasks, SARL outperforms prior label-free RL baselines and even exceeds RL methods with ground truth supervision, with average gains of +9.1% under PPO and +11.6% under GRPO across four math benchmarks, with particularly large improvements on AIME25 (+35.5% with PPO and +44.7% with GRPO). On non-verifiable open-ended tasks, SARL achieves average gains of +34.6% under PPO and +30.4% under GRPO on WildBench across five task categories, outperforming prior label-free RL methods and DPO, which relies on additional preference labels. Beyond strong performance, SARL exhibits substantially lower KL divergence and higher policy entropy, indicating more stable and exploratory training dynamics. Code and data are available at Code Link.
Physics reasoning requires constructing a consistent model of the underlying physical system rather than relying solely on symbolic or formula-based manipulation. Although large language models have shown strong ability in solving math and coding problems, they still struggle with physics problems, as these problems entangle the physical modeling process with mathematical calculations. Humans approach physics by first building a representation of the system before performing calculations. Inspired by this, we introduce a unified framework that distills intermediate representations that explicitly encode the physical modeling process and adopt a two-stage post-training strategy, where supervised fine-tuning establishes structured modeling, and reinforcement learning with rubric-based feedback improves the quality of the modeling process. Experiments on multiple multimodal physics benchmarks show that our approach generally improves physical reasoning performance across different models and datasets. Across PhysReason, PhyX, and SeePhys, physical modeling outperforms GRPO by ~3% on average. showing that explicit physical modeling is an effective strategy for improving physics reasoning in small VLMs.
Ye Zhang, Xuehang Guo, Rui Pan +4
University of Pennsylvania · William & Mary · University of Illinois Urbana-Champaign +3