Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.
Figures & tables
Figure 1: Overview of our results. On the left, we categorize systematic errors in RLVR based on their effect on training dynamics, including delayed training, suboptimal plateaus, and training collapse. On the right, we show representative results of these different error types in a controlled experiment using the Reasoning Gym dataset ( Stojanovski et al., 2025 ) .
Figure 2: Main results across two models (left: OLMo, right: Qwen). We observe the following trends: (i) random noise and systematic FNR lead to delayed training but similar final performance, while systematic FPR leads to plateauing or collapsing behaviors, depending on the trigger used. We provide results with Dr. GRPO and SAPO in Figure 12 .
Figure 3: Output examples under different word-based FPs. The clean and Certainly -triggered runs produce correct arithmetic reasoning, while the python -triggered fails by using incorrect code-based output.
Figure 4: Distribution of tri-gram and final performance (OLMo). The trigrams that remained below 50% FPR are highlighted in red.
Figure 5
Figure 7: Alternation of oracle and noisy verifier. FPR is measured only for the steps with the noisy verifier.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Dataset size
50,000
min_terms
3
max_terms
6
min_digits
3
max_digits
6
min_decimal_places
3
Appendix
Table 1: Training Configuration
Figure 8: An example query used for training.
Figure 9
Figure 11: Examples for learned behaviors in the trigram-based error patterns. For a question 481.869 + 519.413 + 776.7711 - 734.133 , (ground truth: 1043.9201), we provide a representative output example for positive / negative conditional advantage (defined in Section 4.3 ), and low / high frequency. For each caption, we also provide the conditional advantage (CA), frequency (Freq), and the converged reward (R) of the when the trigram is used as the FP trigger.
Figure 12: Results with DAPO, Dr. GRPO, and SAPO.
Figure 13: Multi-seed runs across two models. Across three random seeds, the qualitative dynamics remain stable: clean training improves the reward consistently, relative-error and Certainly -triggered FP plateau, and python -triggered FP collapses.
Figure 14: Systematic but non-deterministic FP. (i) For error patterns that lead to a suboptimal plateau with 100% FPR (relative-error based FP, “Certainly”-triggered FP), the training dynamics improve as error rate decreases. (ii) For triggers that induce collapse (“python”-triggered FP), when p=80% , training is substantially delayed, but it no longer fully collapses as it does in the main experiment with p=100% . Interestingly, the Qwen run initially follows a collapsing trajectory, but the reward begins to recover after around step 100, eventually reaching a suboptimal level within our timeframe. When p is further reduced to 50% , the model no longer exploits the false-positive pattern, and the training curve is close to ideal, with only a slight delay.
Figure 15: Combined FP/FN error. When combining FP and FN rules (denoted as MIX in the legend), we consider two variants for rollouts that satisfy both the FP and FN conditions: treating such rollouts as (i) FNs or (ii) FPs. Compared with the pure-FP setting, we find that adding format-based FN rule has no strong effect, preserving the same qualitative behavior. In contrast, language-based FN rule with variant (i) results in a mere delay for Qwen because the FN rule forces Qwen to generate its answer in Chinese (as analyzed in Section 4.5 of the main paper), which naturally suppresses the generation of the FP trigger "Certainly". For OLMo, however, the same setting leads to complete collapse.
Figure 16: Results on OLMo3-32B. The qualitative observation is consistent with the main experiments on smaller models, where verification errors result in delay (random noise and systematic FN), plateauing (relative-error-based and "Certainly"-triggered FP), or collapse ("python"-triggered FP). We note that the gap between the ideal run and the delayed runs is larger because, due to resource constraints, we trained the model for 250 steps instead of 500.