Compositional Self-Improvement: Promises and Pitfalls
Authors: Changho Shin, Daiwei Chen, Brenden Lake, Frederic Sala, Ramya Korlakai Vinayak
Organizations: Department of Computer Science, Princeton University, Princeton, NJ · Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI · Department of Psychology, Princeton University, Princeton, NJ · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI
Although self-improvement is becoming increasingly important as tasks become more complex, models struggle to generate reliable supervision for harder, out-of-distribution instances. One way to address this challenge is to exploit compositional structure: models often solve simpler subproblems more reliably than full problems, so their subproblem predictions can be composed into supervision for harder ones. We systematically study when this kind of composed supervision supports self-improvement and when it fails. Our controlled experiments identify three failure modes: unreliable subproblem predictions, errors introduced by composition, and limited supervision coverage. Beyond controlled settings, these findings carry over to more realistic coding and reasoning tasks: composed labels are more accurate than direct self-generated labels and improve performance even on compositions longer than those seen during training, while the failure modes remain visible. Further, although these failure modes often keep compositional self-improvement from fully succeeding on its own, a short SFT warmup with compositional self-improvement can bootstrap reinforcement learning on tasks that require composing skills the model already has, particularly when direct exploration rarely succeeds. Together, these findings characterize when compositional self-improvement can provide sufficiently reliable supervision to expand model capabilities.
Figures & tables
Figure 1: Overview of compositional self-improvement. Direct pseudo-labeling can be unreliable on harder instances. Compositional self-improvement constructs targets by combining predictions on simpler subproblems, filters the resulting examples, and trains the model to solve larger problems directly. Updated predictions provide supervision for subsequent rounds.
Table 1: Composition examples for the controlled tasks. Grid routes are joined by the moves through the corridor between rooms, addition concatenates block sums, and max run length keeps the longer run. Filters reject carries across blocks and runs across the join. Blue and purple mark the components; orange marks parts added by the rule.
Figure 2: Compositional self-improvement expands the range of instances solved by the model. Each heatmap shows accuracy across self-improvement rounds. From left to right: grid navigation by the number of rooms, addition by addend digit length, and max run length by string length. In all three tasks, reliable prediction extends beyond the initial model’s reliable range.
Figure 3: Weaker starting models can limit compositional self-improvement, as illustrated by experiments on the grid navigation task. Left: mean accuracy on 2–13 rooms across rounds. Right: accuracy by room count after the final round. The legend gives initial-training updates and one-room accuracy. Curves average three seeds.
Figure 4: Structural errors can undermine self-improvement, although filtering can restore gains in the “addition” tasks including on test sums with boundary carries. Left and middle: accuracy by addend length across rounds without and with filtering. Right: accuracy at each round’s newest lengths (up to 22 – 23 digits), on sums with and without a carry across the block boundary.
Figure 5: Boundary-crossing accuracy in the max run length task. Excess run length is the difference between the longest run in the concatenated string and the longest run in either component.
Task
Component 1
Component 2
Composed parent → label
Filter
Program generation
Output the first 5 lines → head -5
Sort in descending order → sort -r
Step 1: output the first 5 lines. Step 2: sort in descending order. → head -5 | sort -r
Parses as a pipeline
Multi-hop reasoning
Who issued the Royal Proclamation of 1763? → King George III
What year did King George III die? → 1820
What year did the person who issued the Royal Proclamation of 1763 die? → 1820
Answer format; answer found in the passages
Table 2: Composition examples for program generation and multi-hop reasoning.
Figure 6: Compositional self-improvement improves program generation, with less transfer to commands absent from training compositions. The initial model is trained on all individual commands; a command is seen if it also appears in training compositions. (a) Accuracy of direct and compositional self-improvement on test pipelines of seen commands; length 1 tests single commands. (b) Mean accuracy on 2- to 6-command test pipelines whose commands are all seen, include one unseen command, or are all unseen. Round 0 is the initial model; round r trains on pipelines of length r+1 .
Figure 7: Composed supervision improves multi-hop reasoning on MuSiQue. Left: mean accuracy on 2- to 4-hop questions for all sizes, labeled with each initial model’s single-hop accuracy. Middle and right: exact-match accuracy of Qwen3.5-9B by hop count under direct and compositional self-improvement. Round 0 is the initial model, trained on single-hop questions; rounds 1–3 train on 2-, 3-, and 4-hop questions.
Last digit →{\color[rgb]{0.1211,0.4336,0.5469}p_{1}} : return x % 10
Integer to binary string →{\color[rgb]{0.4805,0.3086,0.6602}p_{2}} : return bin(x)[2:]
Step 1: last digit. Step 2: integer to binary string. → return p2 ( p1 (x))
Parses as one function
Table 3: Composition examples for the RL warm-start tasks, showing two functions of longer chains.
Figure 8: A compositional SFT warmup improves subsequent RL on five-operation string transformations and three-function Python programs. CSI+RL first fine-tunes on labels from compositional self-improvement (CSI), then runs RL, using half the token budget of direct RL. Curves show mean test accuracy over three seeds; dotted segments mark the SFT warmup. Token counts include label generation, RL samples, and warmup targets.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Component 1 output:
RRRDRDD
Component 2 output:
RUURRR
Composed target:
RRRDRDD RRDRR RUURRR (corridor actions in the middle)
Appendix
Figure 9: Saved round-1 labels for the algorithmic tasks, with the prompts the model sees. The composed targets and all inputs are transcribed from the saved label files. Grid child routes are recovered by splitting the saved label at the corridor actions; addition component outputs are the model’s saved labels for the same component prompts; max run length component outputs were not saved. All three labels are correct.
Shared instruction block
Use only standard text utilities (such as sort, uniq, grep, cut, tr, head, tail, sed, awk, wc, rev, tac, nl, paste, fold, fmt, cat, column). The text arrives on standard input; do not name any file. Reply with the command on one line and nothing else.
Component 1: model input
Write one Bash command that reads standard input and does the following. Task: Output the first 5 lines of a file
Selected output: head -5
Component 2: model input
Write one Bash command that reads standard input and does the following. Task: Sort a file in descending order
Appendix
Figure 10: Saved Bash composition example label2_00059 from round 1. The shared instruction block shown above is inserted after the opening sentence in each prompt; the remaining prompt text and selected outputs are transcribed unchanged. Execution inputs are passed to the program at evaluation and are not part of the prompt. Line wrapping is for display.
Model input
Saved output
Component 1
Q: func_2(bnqi) = ?
func_2=binq => binq
Component 2
Q: func_15(binq) = ?
func_15=elqt => elqt
Component 3
Q: func_15(elqt) = ?
func_15=hotw => hotw
Component 4
Q: func_4(hotw) = ?
func_4=hOtW => hOtW
Component 5
Q: func_13(hOtW) = ?
func_13=WhOt => WhOt
Appendix
Figure 12: Saved compositional warmup label for strings of depth five. Each input is written as Q: … = ? followed by a new line and A: ; each component input applies one function to the model’s previous output. The parent input is also the prompt of direct RL. All text is transcribed from the saved files, and the label is correct.
Grid navigation
Addition
Max run length
Model
14.2 M-parameter character-level Transformer
Learning rate
5×10−4
5×10−4
5×10−4
Weight decay
0.1
0.1
0.1
Batch size
256
256
256
Initial training
200,000 one-room grids, 30,000 updates
3–7 digits, 50,000 per length, 10,000 updates
Lengths 6–10, 50,000 per length, 15,000 updates
Rounds
12
8
8
Appendix
Table 4: Training setup for the algorithmic tasks.
Program generation
Multi-hop reasoning
Model
Qwen3.5-2B; LoRA rank 16 , α=32
Qwen3.5-2B, 4B, 9B; LoRA rank 16 on attention and feed-forward projections
Table 5: Hyperparameters for the experiments in Section 4 .
Strings
Typed Python
Initial model and compositional warmup
Initial model
14.2 M character Transformer
Qwen3.5-2B, LoRA rank 16 , α=32
Initial training
Chains of 1 – 2 functions
128 updates on single functions
Warmup SFT
24 updates, learning rate 10−4
24 updates, learning rate 10−5
Target tokens per update
8,192
4,096
GRPO
Appendix
Table 6: Hyperparameters for the RL experiments in Section 5 .
Figure 14: Accuracy by instance size across self-improvement rounds for the algorithmic tasks of Section 3.3 , by label source. Runs in a row share the initial model, parents, and schedule. Grid navigation has no filter. The compositional panel for grid navigation and the filtered panels for addition and max run length are the runs in Figure 2 ; the addition panels also appear in Figure 4 .
Figure 15: Addition accuracy by addend length across self-improvement rounds, averaged over two retrained initial models. All runs share these initial models and end at 23 digits; panel widths are proportional to the number of rounds. Length filter: labels shorter than the longest prediction for the same length minus two tokens are discarded ( Lee et al., 2025 ) . Compositional runs use the boundary-carry filter of Table 1 .
Figure 16: Bash outputs on test pipelines of seen commands, by pipeline length. (a) Accuracy. (b) Share of outputs with the correct number of commands, regardless of whether the output is right, and the initial model’s share of single-command outputs. (c) Accuracy on pipelines whose every step the same checkpoint writes correctly alone (see Setup). Panels (b) and (c) pool the four seeds.
Figure 17: Bash accuracy by pipeline length and self-improvement round for test pipelines with no, one, or all commands held out. The same checkpoints are evaluated on three test sets, which differ in how many of their commands were excluded from training pipelines. Length 1 is the single-command test on the 89 seen commands (top) or the 30 held-out commands (middle and bottom). Cells average four seeds; round 0 is the shared initial model.
Figure 18: MuSiQue results for all three model sizes. Cells show direct-answer exact-match accuracy (%), averaged over three seeds. Top: direct self-improvement. Bottom: compositional self-improvement. The 9B panels repeat Figure 7 . After three rounds, compositional self-improvement exceeds direct self-improvement on 2- to 4-hop questions by 15.5 , 11.3 , and 17.1 points on average for 2B, 4B, and 9B, respectively.
Task
Labels
Kept
Wrong labels ( ρ )
Summed component errors ( Σ^ )
Rule errors ( ξ )
Wrong with correct components
Correct with a wrong component
Grid navigation
2 rooms
100
0.9
0.9
0.0
0.0
0.0
2–13 rooms
100
3.5
3.5
0.0
0.0
0.0
Bash
length 2
100
8.8
9.1
0.0
0.0
0.1
length 4
100
30.3
32.3
0.0
0.0
0.3
Strings
depth 5
100
23.0
26.8
0.0
0.0
0.0
Typed Python
depth 3
100
5.9
7.4
0.0
0.0
1.6
Appendix
Table 7: Where the errors of composed labels come from (%), measured on the kept labels of one round. Wrong labels: share of labels that are wrong. Summed component errors: share of labels with a wrong answer in each component, summed over components. Rule errors: share of parents whose label would be wrong even if every component answer were correct. The last two columns count labels that are wrong although every component is correct, and labels that are correct although some component is wrong. Symbols refer to Proposition B.1 . Addition and max run length logged no component answers, so only wrong labels and rule errors are available. Grid navigation rows cover all labels of rounds 1 and 12 (2 and 2–13 rooms), while Figure 19 reports only the newest room count.
Model
Quantity
Round 1 (2 hops)
Round 2 (3 hops)
Round 3 (4 hops)
2B
Compositional labels correct
44.8
42.3
49.9
Direct labels correct
28.9
21.6
24.3
Step error, step 1 to final
12.9, 58.3
15.9, 27.2, 61.8
19.7, 32.6, 33.8, 55.9
Wrong labels, first error at final step
83.4
60.7
39.4
Correct labels with a wrong step
6.8
15.9
30.8
4B
Compositional labels correct
60.5
63.1
65.6
Appendix
Table 8: Training labels in the MuSiQue experiment (%), averaged over three seeds. Label accuracy is scored with the official answer aliases. Step error gives, for each step of the composed labels in order, the share whose answer differs from the released step answer (scored without aliases); the last value is the final step. The last two rows give the share of wrong composed labels whose first wrong step is the final step, and the share of composed labels that are correct although some step answer differs from the released one. Label acceptance never uses reference answers.
Initial-training updates
One-room accuracy
Mean child error
ρ , 2 rooms
ρ , 13 rooms
400
28.4
76.3
94.3
99.4
1,500
58.3
44.1
68.3
93.7
2,500
73.7
29.1
49.7
86.0
3,000
79.9
21.4
38.1
77.1
6,000
90.4
9.2
17.5
48.5
8,000
95.0
5.3
10.3
32.2
Appendix
Table 9: Composed-label error in the grid seed-accuracy study ( Figure 3 ; %), averaged over three seeds. One-room accuracy is the initial model’s test accuracy. Mean child error is Σ^/2 on the 2-room labels of the first round. In every cell, all wrong labels have at least one wrong child route and the rule error is 0 .
Task
Test instances
Initial, direct
Initial, decomposed
Final round, direct
Grid navigation
2–13 rooms
0.0
88.6
80.9
13 rooms
0.0
76.5
47.0
Addition
8–23 digits
0.0
23.2
93.7
23 digits
0.0
4.0
80.0
Max run length
11–83 symbols
59.8
87.9
97.6
83 symbols
37.0
81.0
94.0
Appendix
Table 10: Accuracy (%) of the initial model with direct inference and with decomposed inference, and of the final compositional model with direct inference. Decomposed inference receives the decomposition of each test instance, which direct inference does not. Rows with a size range average the accuracy per size. Bash values average four seeds for the final model; the initial model is shared. MuSiQue values average three seeds and use the 2,417 development questions of Figure 7 .
Task
Initial, direct
Initial, decomposed
After warmup
After warmup and RL
Strings, depth 5
0.0
77.3
11.5
19.2
Strings, depth 4
0.2
85.4
38.7
50.1
Length-preserving, depth 4
0.0
90.0
47.5
61.7
Typed Python
3.3
92.2
22.1
55.6
Appendix
Table 11: Test accuracy (%, 512 parents) in the RL warm-start experiments of Section 5 and Section C.9 . The initial model is the shared initial model; the warmup and RL columns average three seeds, after the warmup and at the last checkpoint ( 40 M tokens for strings, 524 K for Python).
Figure 19: Accuracy of the training labels for the instance size added in each round, audited against reference answers after training. Grid navigation has no filter.
Label source
Round 1
Round 2
Round 3
Compositional
90.6
83.6
80.6
Direct
7.9
2.0
0.5
Appendix
Table 12: Accuracy (%) of training labels. (a) Bash, by self-improvement round, averaged over four seeds. (b) The 512 labels of the single compositional warmup in each RL configuration.
Figure 20: Test accuracy on 512 held-out problems for two depth-four string configurations: (a) all 21 functions and (b) the 11 length-preserving functions. Results are averaged over three seeds. The dotted segment denotes the compositional SFT warmup; token accounting follows Figure 8 .
Self-improvement training, where models learn from self-generated solutions, promises sustained capability gains but suffers from a pervasive failure mode: across multiple rounds, compounding reasoning errors cause accuracy to stall or degrade. We trace this drift to standard filtering criteria that retain solutions based solely on final answer correctness, which lets lucky guesses (correct answers with flawed reasoning) contaminate the training data. We propose Verified Self-Improvement (VSI), a framework that conditions data retention on step-level structural integrity rather than just the final output. VSI validates solutions by recomputing arithmetic steps via a computer-algebra library (sympy), checking intermediate consistency, and enforcing domain constraints. Evaluating VSI on GSM8K with Qwen3-4B-Thinking across 5 rounds of self-improvement against four baselines (no verification, outcome verification, majority voting, and VSI with DPO) shows that VSI rejects approximately 34% of correct-answer solutions, successfully isolating lucky guesses. This cleaner training signal drives sustained accuracy gains across all rounds (80.5% to 91.0%), whereas outcome verification plateaus and unverified training collapses. Finally, converting VSI checks into DPO preference pairs trains the model to distinguish sound reasoning from lucky answers, boosting reward accuracy from 46% to 63%. VSI offers a simple, reproducible recipe for robust self-improvement whenever automated reasoning checks are available.
Does reinforcement learning teach language models new capabilities, or merely amplify ones acquired during pre-training? The literature offers evidence for both: pass@k studies suggest RL reweights latent behaviours while some controlled composition studies demonstrate gains beyond simple amplification. We argue that this question has no answer independent of the capability state of the base model, and propose a two-dimensional state space for post-trainability. The first axis, effective full-solution support, measures whether successful trajectories can be sampled under a feasible exploration budget. The second axis, atomic primitive coverage, measures whether the constituent skills of a solution can be executed when the burden of composition is removed. Together, these dimensions motivate three regimes -- amplification, composition-limited, and acquisition-limited -- each predicting a distinct empirical signature and calling for a distinct intervention. We propose a prospective measurement protocol and investigate the framework using synthetic composition tasks across three small language models. We find that primitive mastery alone does not guarantee that RL can learn a composition. Checkpoints that execute every primitive reliably can fail to improve when complete solutions are never sampled during training, whereas modest composition supervision that establishes nonzero support enables substantial RL gains. The framework reframes mid-training as preparation for post-training: its role is to move models into states from which RL can operate effectively.
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
Siqi Zeng, Andre N. Assis, Rowan Wang
University of Illinois · Constellation · Anthropic