Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
Organizations: The Hong Kong University of Science and Technology · Kuaishou Technology · Tsinghua University
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training method for improving the reasoning abilities of Large Language Models (LLMs). However, existing methods mainly apply uniform optimization constraints across all tokens, ignoring their heterogeneous roles. Prior work shows that high-entropy tokens are closely tied to reasoning, while low-entropy tokens primarily encode factual knowledge, and recent approaches attempt to exploit this distinction by isolating token updates via masking or asynchronous training. We argue that retaining training signals across token types can lead to more effective learning. To address this, we propose Archer, an entropy-aware RLVR framework with dual-token constraints that preserves joint optimization while modulating update strength across token types. Our method introduces response-level entropy normalization for stable token classification and applies differentiated clipping ranges and KL regularization to encourage exploration on reasoning tokens while preserving knowledge tokens. Experiments on mathematical reasoning and code generation benchmarks show that Archer consistently outperforms strong baselines across multiple model scales, improving both pass@1 and pass@K performance. These results highlight the importance of respecting sequence-level dependencies when designing fine-grained RL optimization strategies for LLMs. The code is available at https://github.com/wizard-III/ArcherCodeR.
Figures & tables
| Method | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg. | ||||||
| avg@64 | pass@64 | avg@64 | pass@64 | avg@64 | pass@64 | avg@4 | pass@4 | avg@8 | pass@8 | avg@4 | pass@4 | ||
| DeepSeek-R1-1.5B | 30.6 | 80.0 | 23.5 | 63.3 | 70.7 | 100.0 | 83.6 | 92.4 | 27.6 | 48.2 | 44.6 | 59.4 | 46.8 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{{ {\lxSVG@setlinewidth{\the\pgflinewidth}\lx@inpgf@ignorespaces\lxSVG@setdash{0.0pt}\lx@inpgf@ignorespaces\lxSVG@roundcap\lx@inpgf@ignorespaces\lxSVG@roundjoin\lx@inpgf@ignorespaces{} {}{}{} {}{}{} \lxSVG@stroke}} }{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} GRPO | 42.1 | 80.0 | 28.6 | 56.7 | 80.3 | 97.5 | 87.6 | 94.6 | 29.2 | 46.3 | 53.2 | 65.8 | 53.5 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} DeepScaleR-1.5B | 42.0 | 83.3 | 29.0 | 63.3 | 81.3 | 100.0 | 87.7 | 93.6 | 30.3 | 51.1 | 50.7 | 61.0 | 53.5 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} FastCuRL-1.5B-V3 | 48.1 | 80.0 | 32.7 | 60.0 | 86.4 | 95.0 | 89.8 | 94.0 | 33.6 | 50.0 | 55.3 | 64.3 | 57.7 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} Nemotron-1.5B | 48.0 | 76.7 | 33.1 | 60.0 | 86.1 | 97.5 | 90.6 | 93.6 | 35.3 | 47.8 | 59.2 | 66.8 | 58.7 |
| Method | LCB v5 (2024.08.01-2025.02.01) | LCB v6 (2025.02.01-2025.05.01) | Avg. | ||
| avg@8 | pass@8 | avg@16 | pass@16 | ||
| DeepSeek-R1-1.5B | 16.7 | 29.0 | 17.2 | 34.4 | 17.0 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} GRPO | 26.0 | 40.5 | 27.6 | 43.5 | 26.8 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} DeepCoder-1.5B | 23.3 | 39.1 | 22.6 | 42.0 | 23.0 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} Nemotron-1.5B | 26.1 | 35.5 | 29.5 | 42.8 | 27.8 |
| \mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\hskip 0.325pt\lower-5.32501pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {{\lx@inpgf@ignorespaces}{}{{}}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}{}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.65pt} \lx@inpgf@ignorespaces{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 L 0 -4.7 C 0 -5.93 0.99 -6.92 2.21 -6.92 L 7.45 -6.92}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 7.45 -6.92)} \lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke-width=0.52pt} \lxSVG@begingroup@{stroke-dasharray=none,stroke-dashoffset=0.0pt} \lxSVG@begingroup@{stroke-linecap=round} \lxSVG@begingroup@{stroke-linejoin=round} \lxSVG@drawpath@unclipped{M -1.97 2.63 C -1.81 1.64 0 0.16 0.49 0 C 0 -0.16 -1.81 -1.64 -1.97 -2.63}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}} Archer-Code-1.5B | 29.4 | 43.7 | 30.2 | 45.8 | 29.8 |
| Method | Avg. |
| Archer (masking) | 66.0 |
| Archer | 67.1 |
| AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg. | |
| 0.7 | 50.7 | 42.6 | 91.2 | 95.3 | 50.6 | 71.0 | 66.9 |
| 0.8 | 51.4 | 43.1 | 91.0 | 95.1 | 51.2 | 71.6 | 67.1 |
| 0.9 | 48.3 | 39.9 | 90.9 | 95.0 | 50.4 | 70.3 | 65.8 |
| KL Weight | LiveCodeBench v5 (avg@8) |
| 0.0 | 26.6 |
| 0.001 | 29.4 |
| 0.005 | 26.2 |
| LiveCodeBench v5 (avg@8) | ||
| Varying Low-Entropy Token Clip | ||
| 0.1 | 0.4 | 24.6 |
| 0.2 | 0.4 | 28.7 |
| 0.3 | 0.4 | 26.0 |
| Varying High-Entropy Token Clip | ||
| 0.2 | 0.2 | 27.7 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Training Steps | Stages | GPU Hours |
| Math RL | |||
| DeepScaleR-1.5B | 1750 | 3 | 3,800 A100 |
| FastCuRL-1.5B-V3 | 2620 | 5 | — |
| Nemotron-1.5B | 2500 | 8 | 16,000 H100 |
| Archer-Math-1.5B | 520 | 1 | 1,900 H800 |
| Code RL | |||
RL for Language Models
Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.
See all 460 papers