ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
Authors: Dhruv Dixit, Khushi Patel, Congcong Xu, Min Song
Organizations: Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ - 07030, United States · Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ - 07030, United States
Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce ARCHER: Action-trace Reinforcement for Calculator-grounded Hierarchical Executable Reasoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.
Figures & tables
Figure 1: Overview of ARCHER. Modules in Blue are trainable components, Light Gray are non-trainable or deterministic, Orange are Reward or evaluation, and Green are Data/traces/observations.
Backbone
Method
GSM-8K
SVAMP
MM
MATH
AIME-24
AIME-25
MATH-500
PolyMATH
Tool-Augmented Baseline Framework Comparisons
Qwen3-1.7B
CoT-FSP
69.00
67.71
41.03
57.41
20.00
16.67
46.29
23.81
CoT-FT
72.77
70.48
42.97
62.90
26.67
20.00
51.98
26.39
Toolformer
77.13
74.45
45.44
65.79
30.00
26.67
58.12
29.77
CoA
79.54
77.03
48.13
71.04
36.67
36.67
63.81
33.12
ARCHER
83.72
81.53
51.58
78.06
46.67
43.33
71.36
37.93
Table 1: Comparison of ARCHER with tool-augmented baseline frameworks on five standard backbones and with Vanilla and CoA variants on two reasoning-specialized backbones across eight mathematical reasoning benchmarks. Boldface indicate best values and underlines indicate second-best values. All values are percentages.
Figure 2: Training dynamics study of ARCHER.
Metric
Falcon-H1R-7B
ARCHER w/o HiGRA
ARCHER
Acc. ↑
77.40
79.06
85.96
Tool Score ↑
0.00
0.46
0.76
Tool Recall ↑
0.00
0.58
0.79
Proc. Score ↑
0.52
0.67
0.79
Valid Trace ↑
41.2
68.7
82.4
Obs. Cons. ↑
0.00
0.84
0.93
Table 2: Stage-wise executable diagnostics on GSM-8K with Falcon-H1R-7B backbone. Accuracy and Valid Trace are percentages; all other metrics are reported on [0,1] . Obs. Cons. denotes observation consistency.
Model Variant
Acc. ↑
Tool Score ↑
Proc. Score ↑
Falcon-H1R-7B
77.40
0.00
0.52
CoT-SFT
78.12
0.03
0.55
ARCHER w/o HiGRA
79.06
0.46
0.67
ARCHER w/ outcome-only GRPO
82.35
0.55
0.69
ARCHER w/ flat micro-reward GRPO
83.10
0.64
0.72
ARCHER w/o tool reward
81.74
0.51
0.76
Table 3: Training and reward ablations on GSM-8K with Falcon-H1R-7B backbone. Training-target and HiGRA reward ablations. Accuracy is a percentage; all other metrics are reported on [0,1] .
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Action
Role
READ
Restates or identifies the relevant information in the problem.
PLAN
Specifies the next reasoning step or decomposition.
CALL_CALC
Submits an arithmetic expression to the calculator runtime.
OBSERVE
Records the deterministic output returned by the runtime.
VERIFY
Checks whether the intermediate or final result is consistent.
ANSWER
Emits the final normalized answer.
Appendix
Table A.1: ARCHER action vocabulary. The model generates reasoning actions and calculator-call payloads; calculator observations are produced by the runtime.
Metric
Falcon
ARCHER w/o HiGRA
ARCHER
Acc. ↑
77.40
79.06
85.96
NED act↓
0.48
0.29
0.21
NED op↓
0.55
0.34
0.26
Valid Trace ↑
41.2
68.7
82.4
Tool Prec. ↑
0.00
0.71
0.83
Tool Rec. ↑
0.00
0.58
0.79
Appendix
Table B.1: Structural and tool-use diagnostics on GSM-8K. TRACE substantially improves trace validity and tool use over the base model, while HiGRA further improves both executable structure and tool faithfulness.
Metric
Falcon
ARCHER w/o HiGRA
ARCHER
No Error ↑
77.40
79.06
85.96
Early Error ↓
12.3
8.64
4.84
Tool-Step Error ↓
4.9
7.1
4.3
Post-Tool Error ↓
5.4
5.2
4.9
Mean First-Error Pos. ↑
0.38
0.51
0.64
Appendix
Table B.2: First-error position on GSM-8K. TRACE shifts errors later in the trajectory, and HiGRA further reduces early and tool-step failures.
Reward
Corr. ↑
AUC ↑
Rout
1.00
1.00
Rtool
0.54
0.78
Rproc
0.37
0.69
Ralign
0.29
0.63
RHiGRA
0.68
0.84
Appendix
Table B.3: Association between reward-family scores and final-answer correctness. Auxiliary rewards provide useful but incomplete signals, while the combined HiGRA reward is the strongest non-trivial predictor.
Variant
Acc.
Tool
Proc.
Valid
Obs.
Falcon-H1R-7B
77.40
0.00
0.52
41.2
0.00
CoT-SFT
78.12
0.03
0.55
44.6
0.05
TRACE-SFT
79.06
0.46
0.67
68.7
0.84
Outcome-only GRPO
82.35
0.55
0.69
71.8
0.86
Flat micro-reward GRPO
83.10
0.64
0.72
75.3
0.88
ARCHER
85.96
0.76
0.79
82.4
0.93
Appendix
Table B.4: Detailed objective comparison on GSM-8K. Executable trace supervision is stronger than free-form CoT-SFT, and grouped HiGRA reward optimization outperforms outcome-only and flat micro-reward alternatives.
Compiler Output
Count
Percent
ok
6,982
93.43
calc_only
118
1.58
no_calc _blocks
326
4.36
unsupported _exec
47
0.63
Total prepared examples
7,473
100.0
Appendix
Table B.5: TRACE compiler status counts on the GSM-8K training split. Most examples are successfully compiled, while a small fraction is retained with metadata for transparency.
Sample A: Random
Construction: Fixed-seed random draw from the GSM-8K test set without filtering.
Bias: Unbiased.
Use: Estimate error proportions and report aggregate statistics.
Sample B: Stratified
Construction: Stratified by answer correctness, tool necessity, and reasoning length.
Bias: Diagnostic.
Appendix
Table E.1: Analysis samples used for manual inspection and reward auditing.
Error label
Definition
Typical diagnostic signal
Primary reward family affected
reasoning _error
The model decomposes the problem incorrectly, selects the wrong operation, misses a relevant quantity, or makes an invalid inference before or independent of tool execution.
Table G.11: Dataset splits and example counts used in the experiments. Only the official GSM-8K training split is used for training. All other benchmarks are used exclusively for evaluation using their official test partitions or complete released challenge sets.
Library
Version
Python
3.10.20
datasets
5.0.0
huggingface-hub
1.21.0
mlx
0.31.2
mlx-lm
0.31.3
matplotlib
3.10.9
Appendix
Table G.12: Software library versions used in the ARCHER-RL codebase.
Tool-integrated reasoning (TIR) has emerged as a dominant paradigm for mathematical problem solving in language models, combining natural language (NL) reasoning with code execution. However, this interleaved setup has three key limitations: code often acts as a post-hoc verifier, intermediate NL computations are error-prone, and NL and code play overlapping rather than clearly distinct roles. We propose ThinC (Thinking in Code), a framework in which code itself serves as the reasoner rather than as a tool invoked by NL. A ThinC trajectory begins with a brief NL planning step, after which all reasoning unfolds through code blocks connected only by their execution outputs. We distill 12.2k code-centric trajectories from a teacher model and train ThinC-1.7B and ThinC-4B with supervised fine-tuning followed by reinforcement learning. ThinC-4B consistently outperforms every TIR baseline on five competition-level math benchmarks and even surpasses the much larger Qwen3-235B-A22B-Thinking. Further analysis shows that ThinC reasons through code: 99.2% of its final answers are grounded in interpreter output, and the model recovers reliably from code execution failures without intermediate NL reasoning. Our code and models will be released soon.
Hyeon Hwang, Jiwoo Lee, Jaewoo Kang
Korea University · Korea University, AIGEN Sciences
Large language models (LLMs) demonstrate strong chain-of-thought (CoT) reasoning abilities, while smaller models (<= 3B parameters) significantly underperform on multi-step reasoning tasks. Based on empirical analyses of the Qwen-2.5 model family on math reasoning benchmarks, we find that more proficient reasoning is associated with fewer reasoning steps but higher information density per step, a property we term Dense Reasoning. Motivated by this observation, we propose DenseSteer, a training-free inference-time steering framework that enhances small-model reasoning by modulating internal representations toward dense reasoning patterns. Experiments show that our method yields consistent accuracy improvements without increasing token-level Negative Log-Likelihood, highlighting dense reasoning as an effective structural approach to mathematical problem solving.
Yang Ouyang, Shuhang Lin, Jung-Eun Kim
1North Carolina State University · 2Rutgers University
Knowledge distillation from powerful reasoning models is widely used to improve Small Language Models (SLMs) on mathematical reasoning, often assuming that traces with higher reward model scores provide more useful supervision. We identify a counterintuitive Quality-Utility Paradox in mathematical reasoning distillation. Data refined or synthesized by a stronger Oracle obtains higher perceived quality according to reward models, yet consistently underperforms traces generated by the SLM itself and selected through rejection sampling across Qwen2.5, LLaMA-3, and DeepSeek families. Our analysis shows that Oracle refinement couples logical repair with distributional drift away from the SLM's native reasoning distribution. This drift increases the learner's adaptation cost and can outweigh the benefit of improved reasoning logic. To test this mechanism, we introduce Style-Aligned Refinement, which preserves the native trajectory of the SLM while retaining logical repair from the Oracle. This intervention lowers adaptation cost and restores downstream utility. These findings suggest that effective mathematical reasoning distillation should jointly optimize perceived solution quality and learner-data compatibility, rather than relying solely on reward-model scores. The datasets and code are available at https://github.com/Dracoqhl/Quality-Utility-Paradox.
Haolong Qian, Xianliang Yang, Yinuo ma +6
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Microsoft Research Asia, Microsoft, Beijing, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China +1