Outside the Purview of the Reward: Capability Drift in Multilingual Post-Training
Abstract
Post-training can improve rewarded behavior while weakening capabilities users still need. We study two GRPO branches from a shared supervised checkpoint in a multilingual mixture-of-experts model. The released branch gains 36.07 points on MATH but loses 19.32 F1 points on XQuAD-Indic, while XQuAD-English improves under the same scoring rule. An independently trained sibling adds Indic instruction following and language-aware shaping. It recovers the regressing Indic task scores and preserves MATH, but does not restore script compliance and trades off competition mathematics and code. In a later round, judge training is followed by a sharp extraction decline from the Indic branch; reward-free distillation from its earlier checkpoint restores most of the lost F1. A MATH score decomposition associates most of the measured gain with increased boxed-answer emission; accuracy among boxed answers changes little. These results show why benchmark scores should be read alongside the response forms they reward.
Figures & tables
| Benchmark | Input | Passage | Recovered | Output script | Grader | |
|---|---|---|---|---|---|---|
| language | supplied | from it | constrained | |||
| MATH | En | no | no | no | judge, final answer | |
| GPQA-Diamond | En | no | no | no | exact match, option | |
| IFBench | En | no | no | no | constraint checker | |
| IFEval | En | no | no | no | constraint checker | |
| MMLU-Pro | En | no | no | no | exact match, option |
| Benchmark | SFT-Init | RL-Base | RL-Indic | |
| Indic regressions | ||||
| XQuAD-Indic (EM) | 18.84 | 12.86 | 32.00 | |
| IndicIFEval-ground | 40.52 | 33.05 | 60.00 | |
| IndicQA (EM) | 12.77 | 12.22 | 23.46 | |
| IFEval-Hindi | 52.71 | 50.59 | 72.29 | |
| Other reasoning and instruction-following tasks | ||||
| Measurement | SFT-Init | RL-Base |
|---|---|---|
| Boxed-answer emission | 55.2 | 91.6 |
| Accuracy given a boxed answer | 91.1 | 92.6 |
| Accuracy given an unboxed answer | 1.5 | 11.4 |
| Component of pooled gain | Points | Share |
| Boxed-answer emission rate | 33.13 | 95.2% |
| Accuracy given a boxed answer | 1.39 | 4.0% |
| Checkpoint | Span | Script | EM | F1 |
|---|---|---|---|---|
| RL-Base | 14 | – | 12.65 | 17.17 |
| RL-Indic | 18 | 80.0 | 32.00 | 50.44 |
| Judge-B , step 50 | 153 | 43.7 | 3.54 | 15.10 |
| Judge-B+ , step 40 | 210 | 11.2 | 1.87 | 7.67 |
| Judge-B+ , step 50 | 83 | 53.1 | 8.49 | 22.79 |
| RL-OPD , step 20 | 26 | 83.2 | 33.52 | 47.13 |
| Round | Checkpoint | EM | F1 |
|---|---|---|---|
| 1, verifiable rewards | RL-Indic | 32.00 | 50.44 |
| 2, judge reward | Judge-I+ , step 20 | 1.91 | 17.67 |
| 3, no reward | RL-OPD , step 20 | 33.52 | 47.13 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Earlier delta | Corrected delta |
|---|---|---|
| MMLU | ||
| CrossSum-Indic (chrF) | ||
| HumanEval+ | ||
| AIME 2025 |
| Setting | Value |
|---|---|
| Export point | partway through the final pass, verified from export metadata |
| Scheduled sequence positions | 80.53B ( ), including padding |
| Approximate loss-bearing positions | 15B (18.2%, assistant-only mask) |
| Trainable-rate measurement | 95,515 / 524,288 positions over four packed rows |
| Intra-pack packing efficiency | 99.98% |
| Prepared packed capacity | 92.12B positions across 702,824 sequences |
| Benchmark | Metric | Final SFT | RL ckpt | (RL SFT) |
|---|---|---|---|---|
| ckpt | step 40 | |||
| Math & Reasoning | ||||
| MATH | exact match | 48.69 | 84.76 | |
| MATH-500 | exact match | 48.80 | 84.00 | |
| GSM8K | exact match | 85.75 | 94.31 | |
| MGSM-English | exact match | 91.60 | 98.40 | |
| Setting | Value |
|---|---|
| Initialization | SFT-Init |
| Algorithm | GRPO with a leave-one-out baseline and per-group Bessel-corrected standard-deviation normalization, divisor , no floor |
| Resolved estimator flags | use_leave_one_out_baseline=true , normalize_rewards=true |
| Generations per prompt | 16 |
| Dynamic sampling | enabled, zero-variance groups resampled |
| Sampling temperature | 1.0 |
| Setting | Value | Pool | Rows | Share |
|---|---|---|---|---|
| Added environment | indic_if | Mathematics | 6,192 | 40.5% |
| Reward | Indic instruction following | 3,179 | 20.8% | |
| Malformed / abandoned | Structured outputs | 1,932 | 12.6% | |
| Advantage std. floor | , with 44% of indic_if groups below it | Workplace assistant | 1,804 | 11.8% |
| Steps / rows / validation | 40 / 15,300 / 256 | English guardrail | 1,509 | 9.9% |
| Recorded rollouts | 251,906 | MCQA | 684 | 4.5% |
| Directive | Initial in-script rate | Rate after 20 steps |
|---|---|---|
| Explicit Hindi name | 38.4% | 90.5% |
| Implicit or referential wording | near zero | |
| Environment | Script-free share | |||
|---|---|---|---|---|
| Structured outputs | 43.4% | 0.3% | ||
| Workplace assistant | 19.3% | 7.5% | ||
| MCQA | — | — | — | |
| Mathematics with judge | — | — | 10.1% | |
| Aggregate | (SE 0.0095) | |||
| 95% bootstrap CI | ||||
| Retained file | Mean reward | Logged reference |
|---|---|---|
| Early-level file A | 0.324 | 0.333 (steps 1–8) |
| Early-level file B | 0.312 | 0.333 (steps 1–8) |
| Late-level file | 0.717 | 0.698 (steps 24–49) |
| Sample | Mixed-script groups | |
| Early file, first chunk | 232 | |
| Early file, middle chunk | 224 | |
| Early file, final chunk | 175 | |
| Late-level retained file | — | 2 |
| Fresh sample, step 20 | — | 1.7% |
| Fresh sample, step 30 | — | 0.9% |
| Variant | Paired groups | |
| Primary answer-span filter | 1,148 | |
| Script-bearing-character filter | 1,116 | |
| No length filter | — | |
| Second seed, truncated sample | 554 | |
| Change from dropping versus absorbing ambiguous spans | 0.0009 | |
| Unfiltered mathematics differential | ||
| Step | 0 | 20 | 60 | 100 | 140 | 160 |
|---|---|---|---|---|---|---|
| Pairwise accuracy | 0.661 | 0.810 | 0.813 | 0.820 | 0.831 | 0.812 |
| Format validity | 0.996 | 0.999 | 0.998 | 1.000 | – | – |
| Total variation distance | 0.229 | 0.305 | 0.108 | 0.083 | 0.173 | 0.156 |
| RL-Indic | Judge-I 20 | Judge-I+ 20 | Judge-I+ 50 | |
|---|---|---|---|---|
| IndicIFEval-ground, strict | 60.00 | 62.11 | 63.96 | 65.06 |
| IndicIFEval-translated, strict | 63.27 | 59.31 | 59.14 | 58.91 |
| IFBench, strict | 50.33 | 44.00 | 45.33 | 40.67 |
| AIME 2025, pass@1 | 65.42 | 67.29 | 67.08 | 66.04 |
| XQuAD-Indic, F1 | 50.44 | 19.34 | 17.67 | 19.08 |
| XQuAD-Indic, EM | 32.00 | 3.79 | 1.91 | 2.85 |
| Aggregation | RL-Base | RL-Indic | Difference |
|---|---|---|---|
| Shared metrics, QA exact match (33) | 0.661188 | 0.661233 | |
| QA rows scored by F1 instead | — | — | |
| Non-instruction-following subset (27) | — | — |
| Benchmark | SFT | RL-Base | RL-Indic | ||
| Math & Reasoning | |||||
| MATH | 48.69 | 84.76 | 86.76 | ||
| MATH-500 | 48.80 | 84.00 | 87.60 | ||
| GSM8K | 85.75 | 94.31 | 91.58 | ||
| MGSM-English | 91.60 | 98.40 | 97.60 | ||
| MGSM-Indic | 79.60 | 84.60 | 86.40 | ||
RL Post-Training
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
See all 464 papers