ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
Organizations: Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ - 07030, United States · Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ - 07030, United States
Abstract
Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce ARCHER: Action-trace Reinforcement for Calculator-grounded Hierarchical Executable Reasoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.
Figures & tables
| Backbone | Method | GSM-8K | SVAMP | MM | MATH | AIME-24 | AIME-25 | MATH-500 | PolyMATH |
|---|---|---|---|---|---|---|---|---|---|
| Tool-Augmented Baseline Framework Comparisons | |||||||||
| Qwen3-1.7B | CoT-FSP | 69.00 | 67.71 | 41.03 | 57.41 | 20.00 | 16.67 | 46.29 | 23.81 |
| CoT-FT | 72.77 | 70.48 | 42.97 | 62.90 | 26.67 | 20.00 | 51.98 | 26.39 | |
| Toolformer | 77.13 | 74.45 | 45.44 | 65.79 | 30.00 | 26.67 | 58.12 | 29.77 | |
| CoA | 79.54 | 77.03 | 48.13 | 71.04 | 36.67 | 36.67 | 63.81 | 33.12 | |
| ARCHER | 83.72 | 81.53 | 51.58 | 78.06 | 46.67 | 43.33 | 71.36 | 37.93 | |
| Metric | Falcon-H1R-7B | ARCHER w/o HiGRA | ARCHER |
|---|---|---|---|
| Acc. | 77.40 | 79.06 | 85.96 |
| Tool Score | 0.00 | 0.46 | 0.76 |
| Tool Recall | 0.00 | 0.58 | 0.79 |
| Proc. Score | 0.52 | 0.67 | 0.79 |
| Valid Trace | 41.2 | 68.7 | 82.4 |
| Obs. Cons. | 0.00 | 0.84 | 0.93 |
| Model Variant | Acc. | Tool Score | Proc. Score |
|---|---|---|---|
| Falcon-H1R-7B | 77.40 | 0.00 | 0.52 |
| CoT-SFT | 78.12 | 0.03 | 0.55 |
| ARCHER w/o HiGRA | 79.06 | 0.46 | 0.67 |
| ARCHER w/ outcome-only GRPO | 82.35 | 0.55 | 0.69 |
| ARCHER w/ flat micro-reward GRPO | 83.10 | 0.64 | 0.72 |
| ARCHER w/o tool reward | 81.74 | 0.51 | 0.76 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Action | Role |
|---|---|
| READ | Restates or identifies the relevant information in the problem. |
| PLAN | Specifies the next reasoning step or decomposition. |
| CALL_CALC | Submits an arithmetic expression to the calculator runtime. |
| OBSERVE | Records the deterministic output returned by the runtime. |
| VERIFY | Checks whether the intermediate or final result is consistent. |
| ANSWER | Emits the final normalized answer. |
| Metric | Falcon | ARCHER w/o HiGRA | ARCHER |
|---|---|---|---|
| Acc. | 77.40 | 79.06 | 85.96 |
| NED | 0.48 | 0.29 | 0.21 |
| NED | 0.55 | 0.34 | 0.26 |
| Valid Trace | 41.2 | 68.7 | 82.4 |
| Tool Prec. | 0.00 | 0.71 | 0.83 |
| Tool Rec. | 0.00 | 0.58 | 0.79 |
| Metric | Falcon | ARCHER w/o HiGRA | ARCHER |
|---|---|---|---|
| No Error | 77.40 | 79.06 | 85.96 |
| Early Error | 12.3 | 8.64 | 4.84 |
| Tool-Step Error | 4.9 | 7.1 | 4.3 |
| Post-Tool Error | 5.4 | 5.2 | 4.9 |
| Mean First-Error Pos. | 0.38 | 0.51 | 0.64 |
| Reward | Corr. | AUC |
|---|---|---|
| 1.00 | 1.00 | |
| 0.54 | 0.78 | |
| 0.37 | 0.69 | |
| 0.29 | 0.63 | |
| 0.68 | 0.84 |
| Variant | Acc. | Tool | Proc. | Valid | Obs. |
|---|---|---|---|---|---|
| Falcon-H1R-7B | 77.40 | 0.00 | 0.52 | 41.2 | 0.00 |
| CoT-SFT | 78.12 | 0.03 | 0.55 | 44.6 | 0.05 |
| TRACE-SFT | 79.06 | 0.46 | 0.67 | 68.7 | 0.84 |
| Outcome-only GRPO | 82.35 | 0.55 | 0.69 | 71.8 | 0.86 |
| Flat micro-reward GRPO | 83.10 | 0.64 | 0.72 | 75.3 | 0.88 |
| ARCHER | 85.96 | 0.76 | 0.79 | 82.4 | 0.93 |
| Compiler Output | Count | Percent |
|---|---|---|
| ok | 6,982 | 93.43 |
| calc_only | 118 | 1.58 |
| no_calc _blocks | 326 | 4.36 |
| unsupported _exec | 47 | 0.63 |
| Total prepared examples | 7,473 | 100.0 |
| Sample A: Random |
|---|
| Construction: Fixed-seed random draw from the GSM-8K test set without filtering. |
| Bias: Unbiased. |
| Use: Estimate error proportions and report aggregate statistics. |
| Sample B: Stratified |
| Construction: Stratified by answer correctness, tool necessity, and reasoning length. |
| Bias: Diagnostic. |
| Error label | Definition | Typical diagnostic signal | Primary reward family affected |
|---|---|---|---|
| reasoning _error | The model decomposes the problem incorrectly, selects the wrong operation, misses a relevant quantity, or makes an invalid inference before or independent of tool execution. | Incorrect plan, wrong equation setup, wrong mathematical relation, or unsupported intermediate conclusion. | , sometimes |
| wrong_tool _input | The model invokes a tool, but the submitted expression or arguments are incorrect. | Calculator call uses the wrong operation, wrong quantity, wrong order of arguments, or incomplete expression. | , |
| wrong_tool _observation | The generated observation is inconsistent with the executable result of the tool call. | The trace records a value that does not match the calculator output for the submitted expression. | |
| missing_tool _call | A tool call is needed according to the gold trace or task structure, but the model does not invoke the tool. | The trajectory performs arithmetic internally or jumps to an answer without an expected calculator call. | , |
| spurious_tool _call | The model invokes a tool when tool use is unnecessary or irrelevant. | Calculator call appears for a simple non-computational step or for an expression unrelated to the problem. | , |
| wrong_after _correct_tool | The tool call and observation are correct, but the model misuses the result in later reasoning or answer synthesis. | Correct calculator output appears in the trace, but the final answer or next reasoning step contradicts it. | , |
| Error label | Definition | Typical diagnostic signal | Primary reward family affected |
|---|---|---|---|
| premature_final _answer | The model terminates before sufficient reasoning, tool use, or verification has occurred. | Final answer is emitted immediately after reading or planning, before required evidence is established. | , |
| format_or _trace_error | The trace is malformed, incomplete, duplicated, or violates the canonical action structure. | Missing final-answer tag, malformed tool-call tag, observation without call, repeated incompatible actions, or unparsable transcript. | , |
| loop_or _verbosity_error | The trajectory is excessively long, repetitive, or trapped in non-productive reasoning. | Repeated plans, repeated calculator calls with no progress, or reasoning that exceeds the rollout budget. | , |
| gold_trace _mismatch | The generated schema or operator pattern diverges substantially from the compiled gold trace, even if some local steps are plausible. | Missing expected action types, wrong operation sequence, or incompatible verify-step structure. | |
| mixed_or _ambiguous | No single failure type is clearly dominant, or several failures are tightly entangled. | Multiple errors occur at nearly the same point, or the first causal failure cannot be determined reliably. | Case dependent |
| Field | Meaning |
|---|---|
| Primary and secondary manual error labels | |
| reasoning_error | Incorrect decomposition, inference, or mathematical reasoning before or independent of tool execution. |
| wrong_tool_input | Tool was called, but the submitted expression or arguments were incorrect. |
| wrong_tool_observation | Tool observation in the trace is inconsistent with executable tool output. |
| missing_tool_call | A needed tool call is absent despite tool necessity in the gold trace. |
| spurious_tool_call | Tool was invoked unnecessarily on an example that does not require it. |
| Bucket | Definition |
|---|---|
| Answer failure | Final answer is incorrect, regardless of upstream cause. |
| Tool omission | A needed tool call is missing. |
| Tool faithfulness failure | Tool input or tool observation is inconsistent with executable tool behavior. |
| Post-tool reasoning failure | Tool use is correct, but later reasoning or answer synthesis is wrong. |
| Premature stopping | Final answer is produced before sufficient evidence or intermediate support. |
| Process failure | Trace is malformed, incomplete, duplicated, or excessively verbose. |
| Manual error label | Expected dominant reward signal |
|---|---|
| reasoning_error | , sometimes |
| wrong_tool_input | and |
| wrong_tool_observation | |
| missing_tool_call | |
| spurious_tool_call | |
| wrong_after_correct_tool | and |
| Variant | Description |
|---|---|
| Falcon-H1R-7B | Base model evaluated without ARCHER training. |
| ARCHER w/o HiGRA | TRACE supervised fine-tuning on compiled executable traces. |
| ARCHER | TRACE initialization followed by HiGRA reward-guided refinement. |
| Action | Implementation role |
|---|---|
| READ | Identify or restate relevant information. |
| PLAN | Specify decomposition or the next reasoning step. |
| CALL_CALC | Submit an arithmetic expression to the calculator. |
| OBSERVE | Insert the deterministic calculator output. |
| VERIFY | Check intermediate or final consistency. |
| ANSWER | Emit the normalized final answer. |
| Compiler setting | Value |
|---|---|
| Include fact extraction action | true |
| Include verification action | true |
| Maximum reasoning span length | 220 characters |
| Repair arithmetic result with runtime output | true |
| Default schema label | arithmetic_word _problem |
| Status | Meaning |
|---|---|
| ok | Reasoning text and calculator spans are both detected. |
| calc_only | Calculator spans are detected with little or no remaining reasoning text. |
| no_calc_blocks | No inline calculator annotations are detected. |
| unsupported_exec | A calculator span cannot be executed by the current runtime. |
| Operation | Supported syntax |
|---|---|
| Addition | + |
| Subtraction | - |
| Multiplication | |
| Division | / |
| Integer division | // |
| Modulo | % |
| Hyperparameter | Value |
|---|---|
| Applicable backbones | All five evaluated backbones |
| Training method | LoRA SFT |
| LoRA rank | 8 |
| LoRA alpha | 16 |
| LoRA dropout | 0.05 |
| Target modules | q_proj , k_proj , v_proj , o_proj |
| Hyperparameter | Value |
|---|---|
| Initial policy | TRACE checkpoint |
| Applicable backbones | All five evaluated backbones |
| Quantized loading | 4-bit enabled |
| Train batch size per device | 1 |
| Gradient accumulation steps | 4 |
| Learning rate |
| Coefficient | Value |
|---|---|
| Tool-use presence | 0.2 |
| Observation presence | 0.2 |
| Observation consistency | 0.6 |
| Spurious tool-use penalty | 0.2 |
| Alignment gate threshold | 0.6 |
| Format score | 0.4 |
| Inference setting | Value |
|---|---|
| Maximum executable steps | 12 |
| Single-action generation limit | 96 new tokens |
| Executable-loop decoding | greedy |
| Active GSM-8K tool | calc |
| Direct generation length | 512 new tokens |
| Direct-generation decoding | greedy |
| Answer type | Normalization |
|---|---|
| Numeric | Remove commas, normalize decimals, compare canonical numbers. |
| Math | Prefer boxed answer or final-answer phrase; otherwise extract the last number. |
| Multiple choice | Compare answer text, choice index, and letter aliases. |
| Text QA | Lowercase, strip punctuation and articles, normalize whitespace. |
| Benchmark | Original dataset structure | Train used | Eval. used | Usage in this work |
|---|---|---|---|---|
| GSM-8K ( Cobbe et al., 2021 ) | 7,473 training and 1,319 test examples | 7,473 | 1,319 | Official training split for TRACE and HiGRA; official test split for evaluation. |
| SVAMP ( Patel et al., 2021 ) | Complete challenge set of 1,000 examples | 0 | 1,000 | Complete original challenge set; evaluation only. |
| MMLU-Math (MM) ( Hendrycks et al., 2020 ) | Official development, validation, and test partitions for five mathematics subjects | 0 | 1,064 | Union of the official test partitions; evaluation only. |
| MATH ( Hendrycks et al., 2021 ) | 7,500 training and 5,000 test problems | 0 | 5,000 | Complete official test split; evaluation only. |
| MATH-500 ( Lightman et al., 2024 ) | Test-only subset of 500 MATH problems | 0 | 500 | Complete released subset; evaluation only. |
| AIME-2024 ( Balunović et al., 2025 ) | 15 AIME I and 15 AIME II problems | 0 | 30 | Complete combined 2024 competition set; evaluation only. |
| Library | Version |
| Python | 3.10.20 |
| datasets | 5.0.0 |
| huggingface-hub | 1.21.0 |
| mlx | 0.31.2 |
| mlx-lm | 0.31.3 |
| matplotlib | 3.10.9 |
LLM Reasoning
Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.
See all 958 papers