How Post-Training Influences Verbalized Algorithmic Computation in Language Models
Organizations: University College London
Abstract
Post-training improves language-model reasoning, but how it changes the use of internal information and verbalized chain of thought (CoT) remains unclear. We study successive SFT, DPO, and RLVR checkpoints of Olmo-3-7B-Instruct on multi-digit addition, using carry values as known intermediate state. Probes recover carry information from internal activations with similar accuracy in SFT and RLVR, although manipulating this information does not reliably control model outputs. In contrast, the usefulness of verbalized reasoning differs: CoTs produced by RLVR yield higher accuracy across receiving checkpoints than those produced by SFT or DPO. We also show that carry-related statements receive higher importance scores after RLVR than matched random controls. DPO shows similar sensitivity to carry-related statements but without comparable accuracy gains. These results suggest that post-training changes the usefulness of written reasoning and sensitivity to its content in meaningful ways, although greater sensitivity does not necessarily coincide with better problem performance.
Figures & tables
| Dataset | Construction | Size used in the paper |
|---|---|---|
| Behavioural addition | Two-, four-, and six-digit addition. The standard set contains four controlled carry structures and a carry-balanced random sample. Separate random samples test delimited operands and reversed answer digits. | 384 standard problems and 768 standard prompt instances; 1,536 prompt instances including the two format conditions. |
| Probe addition | Two-, four-, and six-digit addition, balanced by carry count and divided into disjoint training and test sets. Every problem appears under both prompt conditions. | 384 problems: 192 training and 192 test; 768 prompt instances in total. |
| Paired Free CoT addition | Four- and six-digit problems for which natural reasoning traces are collected from SFT, DPO, and Full. Full is used to choose the initial set; analyses comparing reasoning sources use only problems with a usable trace from every checkpoint. | 384 candidates; 124 problems in the initial set; 77 problems in the three-source subset. |
| On-policy addition | Fresh four-digit Full traces containing an identifiable use of an incoming carry, an earlier statement that establishes the carry, sufficient following context, and matched control spans. | 192 candidates; 51 problems satisfy all analysis requirements. |
| Paired Free CoT subtraction | Four-digit subtraction, balanced across zero, one, two, and three borrows. Full and SFT provide paired reasoning traces; all three checkpoints are evaluated as receivers. | 192 candidates; 55 initially selected problems; 47 problems with usable traces from both sources. |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Paper analysis | Dataset subset | Effective sample |
|---|---|---|
| Behaviour and CoT usefulness ( Figures 2(a) and 2(b) ) | Standard behavioural addition. Accuracy compares paired completed responses; token-budget curves use Free CoT. | 384 problems attempted per prompt condition and checkpoint. The paired accuracy comparison retains 383 SFT, 312 DPO, and 291 Full problems. |
| CoT crossover ( Figure 2(c) ) | Problems with an eligible SFT, DPO, and Full trace. Every source trace is supplied to every receiver. | 77 problems: 45 four-digit and 32 six-digit; 77 observations in each source–receiver cell. |
| Linear probes ( Figure 3 ) | Free CoT instances from the disjoint probe training and test sets; SFT and Full only. | SFT: 192 training and 192 test examples. Full: 125 training and 116 test examples after the completion filter. |
| Probe steering and activation interventions ( Sections E.4 and F ) | Probe steering reuses the probe dataset. Residual interventions use a separate shared Free CoT set. Distributed alignment search and activation patching use controlled carry pairs. | Residual interventions: 100 problems, of which 89 yield eligible cases. Alignment search: 1,024 training, 256 tuning, and 64 evaluation pairs. Activation patching: 16 development and 64 evaluation pairs. |
| Answer emergence and commitment | Paired trajectories use the three-source subset; single-source trajectory summaries use every eligible trace. | 77 problems and 231 traces for paired comparisons; 324 traces for single-source summaries. |
| Semantic importance in addition ( Figure 1(a) ) | Paired SFT/Full traces, divided into reasoning units, with SFT, DPO, and Full as receivers. | 101 problems, 202 traces, and 5,598 reasoning units. |
| Checkpoint | Matched | Answer Only | Free CoT | [95% CI] | Gains | Losses | Exact | Holm |
|---|---|---|---|---|---|---|---|---|
| SFT | 383 | 79.1 | 84.1 | [0.0, 9.9] | 58 | 39 | .0671 | .1341 |
| DPO | 312 | 83.3 | 80.8 | 2.6 [ 8.0, 2.6] | 32 | 40 | .4096 | .4096 |
| Full | 291 | 86.9 | 93.1 | [1.7, 10.7] | 32 | 14 | .0114 | .0341 |
| Quantity | Estimate |
|---|---|
| Common problems | 290 |
| SFT: Answer Only | 86.6% |
| SFT: Free CoT | 89.0% |
| SFT prompt effect | pp |
| Full: Answer Only | 86.9% |
| Full: Free CoT | 93.1% |
| Checkpoint | Split | 2-digit | 4-digit | 6-digit | Total |
|---|---|---|---|---|---|
| SFT | Train | 64 | 64 | 64 | 192 |
| SFT | Test | 64 | 64 | 64 | 192 |
| Full | Train | 64 | 44 | 17 | 125 |
| Full | Test | 64 | 36 | 16 | 116 |
| Target | Token location | Majority baseline | SFT | Full |
|---|---|---|---|---|
| Incoming carry | Prompt final | 57.3 | 71.1 [65.5, 77.4] | 73.8 [69.1, 76.8] |
| Incoming carry | CoT two-thirds | 57.3 | 57.2 [54.0, 61.5] | 60.8 [52.7, 66.8] |
| Incoming carry | Answer digits | 57.0 | 66.9 [61.2, 70.2] | 63.0 [56.8, 66.3] |
| Outgoing carry | Prompt final | 54.8 | 72.2 [69.0, 76.6] | 72.7 [69.6, 75.3] |
| Outgoing carry | CoT two-thirds | 54.8 | 58.0 [53.0, 62.5] | 57.1 [52.4, 61.9] |
| Outgoing carry | Answer digits | 54.8 | 74.9 [65.5, 78.1] | 72.2 [66.4, 75.3] |
| Checkpoint | Problems | Contrasts | Mean [95% CI] |
|---|---|---|---|
| SFT | 73 | 518 | |
| Full | 73 | 522 |
| Receiver | Reasoning source | Correct-answer log probability | Final-answer accuracy (%) | |||
| One-third | Two-thirds | Complete | Two-thirds | Complete | ||
| SFT | SFT | 19 | 89 | |||
| SFT | Full | 22 | 96 | |||
| Full | SFT | 20 | 91 | |||
| Full | Full | 17 | 95 | |||
| Measure | SFT | Full |
|---|---|---|
| Subspace dimensions | 2,137 (52.2%) | 2,102 (51.3%) |
| Tuning interchange | 3.1% | 9.4% |
| Held-out interchange | 1.6% | 12.5% |
| Target digit changed | 1.6% | 14.1% |
| Other digits preserved | 62.5% | 90.3% |
| Layer | Writer reader | Target [95% CI] | Answer [95% CI] |
|---|---|---|---|
| 27 | SFT SFT | ||
| 27 | SFT Full | ||
| 27 | Full SFT | ||
| 27 | Full Full | ||
| 28 | SFT SFT | ||
| 28 | SFT Full |
| Operation | Problems | Units | Units separated before K-means | K-means partition | Median ARI |
|---|---|---|---|---|---|
| Addition | 101 | 5,598 | Punctuation-only formatting units | 11 text clusters; four were merged into mixed/other | 0.660 |
| Subtraction | 47 | 2,783 | Literal borrow mentions and punctuation-only formatting units | 10 text clusters; four were merged into mixed/other | 0.695 |
| Category | Description | Representative unit | Units |
|---|---|---|---|
| Number decomposition | Extracts or aligns operand digits by place value. | “6 (from 221,651)” | 221 |
| Incoming carry application | Explicitly incorporates a carry established in an earlier column. | “ (carry) ” | 195 |
| Staging headers | Announces the next step or column to be processed. | “Step 2: Add the tens place” | 696 |
| Tens-place setup | Introduces or labels the tens-column calculation. | “Tens:” | 163 |
| Carry tracking | Records an output digit and, where present, a newly produced carry. | “Write down 0, carry over 1” | 405 |
| Step-by-step cue | Generic preamble introducing a worked solution. | “Let’s solve the addition step by step” | 168 |
| Category | Description | Representative unit | Units |
|---|---|---|---|
| Subtraction step headings | Announces the place-value subtraction step to be performed. | “Step 3: Subtract the tens place” | 237 |
| Problem heading | Short structural heading introducing the problem. | “Problem:” | 63 |
| Step-by-step cue | Generic preamble introducing a worked subtraction. | “Let’s solve the subtraction step by step” | 79 |
| Place-value headings | Names the current place or place-value stage. | “Hundreds place:” | 116 |
| Result digit assembly | Lists place-labelled result digits or combines them into an answer. | “6 (thousands), 5 (hundreds), 6 (tens), 9 (ones)” | 136 |
| Place-result recording | Records a digit in a named place after a local calculation. | “Write down 0 in the tens place” | 120 |
Language Model Post-Training
Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.
See all 208 papers