Inference-time scaling improves the performance of large language models by sampling multiple responses and evaluating their collective utility. However, many existing approaches obtain these samples by repeatedly sampling from a single policy. In this work, we study if the sample budget at inference time can instead be allocated across a population of policies. We introduce stratified policies, where each of k learned policies contributes one response to a k-sample inference-time objective. We realize this policy population using separate LoRA adapters on a shared language model backbone and optimize it for pass@k. We theoretically motivate the approach by analyzing the linear setting; a population of LoRAs can achieve zero best-of-k approximation error in settings where every single LoRA has strictly positive prediction error. Empirically, we demonstrate that stratified policies consistently improve pass@k over a single-policy baseline across multiple reasoning tasks. Overall, our results highlight how distributing samples across complementary policies can be a more effective use of sample budget than repeated i.i.d. sampling from a single policy at inference time.
Figures & tables
Figure 1: An illustrative example of stratification. With a fixed budget of five responses, i.i.d. sampling from a single policy may repeatedly sample the same solution modes (left). For five modes, the probability of covering all modes is at most 5!/55≈3.84% , which is attained when the policy assigns equal probability to each mode. In contrast, stratified sampling allocates one response to each policy in a learned population, which allows specialized policies to produce complementary responses (right); under perfect specialization, all five modes are covered with probability one.
Figure 2: An illustration of Theorem 1 . Colored segments represent the regional target maps f⋆(x)=Tjx . A rank-1 population fits every region exactly, whereas the best single linear map Bx has an irreducible gap ε⋆>0 .
Figure 3: We evaluate our method on a diverse set of reasoning tasks. These span constructing valid arithmetic expressions, logical deduction, constraint satisfaction, and mathematical word problems.
Figure 4: Pass@5 accuracy of a single policy ( Single ) versus our stratified population of five specialized policies ( Stratified ) across reasoning tasks. Mixture (dashed) samples from the same five policies i.i.d. instead of once from each. Error bars denote standard error across three seeds. Each panel’s y -axis is truncated (kink mark) to make the gap visible. The (renormalized) horizontal bar below each panel shows how much each policy contributes to the population’s unique-only solutions, i.e., problems solved by exactly one policy. Stratified outperforms Single on pass@5 across all four tasks, and each policy member contributes unique solutions to the population.
Figure 5: Pairwise Jaccard similarity between the sets of problems solved by each policy. Lower values indicate that a pair of policies solve mostly disjoint subsets of problems; a value of 1 indicates total overlap. Arithmetic , Knights-&-Knaves , and Sudoku show low overlap; the values for GSM-Symbolic remain ≤0.9 , which still shows non-redundancy in this harder setting.
Figure 6: Distribution, over solved problems, of the number of policies solving each problem under greedy decoding. Colors show the contribution of each policy. For Arithmetic , Knights-&-Knaves , and Sudoku , mass is concentrated at low counts; this indicates that most solved problems are solved by only one or two policies (not the full population). For GSM-Symbolic , there is substantial mass on counts below 5 ; this shows that not all problems are solved by all 5 policies.
Figure 7: (a) Scaling the rank of Single by 5× ( Big-Single ) to match Stratified ’s total trainable parameters still underperforms it. (b) Single is unstable on GSM-Symbolic ; the final checkpoint, Single (Final), achieves a substantially lower score than the pre-collapse checkpoint used elsewhere. Stratified does not exhibit this collapse on any seed. (c) An independently trained ensemble ( Independent ) with matched total training compute underperforms Stratified .
Figure 8: Comparison with UpSkill on Arithmetic . (a) Stratified outperforms UpSkill , which uses an explicit diversity reward. (b, c) Operator distribution of each policy. Stratified policies achieve a comparable level of diversity to UpSkill : different policies focus on different operators.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
LoRA Rank
32 (160 for Big-Single )
Population Size k ( num_generations )
5
Training Batch Size
5 (one prompt per training step)
Training Epochs
1
Optimizer
AdamW (8-bit)
Adam β1 , β2
0.9, 0.99
Appendix
Table 1: GRPO training hyperparameters.
Arithmetic
Knights-&-Knaves
Sudoku
GSM-Symbolic
Method
Pass@1
Pass@5
Pass@1
Pass@5
Pass@1
Pass@5
Pass@1
Pass@5
Single
93.4 ( 3.9 )
95.5 ( 2.8 )
56.8 ( 0.8 )
90.6 ( 0.6 )
40.9 ( 0.3 )
74.2 ( 0.6 )
52.1 ( 4.0 )
85.5 ( 2.4 )
Stratified , adapter 1
62.9 ( 3.7 )
–
36.6 ( 6.4 )
–
34.5 ( 0.8 )
–
56.2 ( 0.5 )
–
Stratified , adapter 2
40.2 ( 0.4 )
–
32.5 ( 3.2 )
–
34.8 ( 1.7 )
–
54.9 ( 1.1 )
–
Stratified , adapter 3
23.5 ( 4.1 )
–
41.4 ( 0.9 )
–
35.5 ( 0.5 )
–
56.9 ( 0.7 )
–
Stratified , adapter 4
20.1 ( 1.1 )
–
45.0 ( 0.2 )
–
34.0 ( 0.7 )
–
56.9 ( 0.6 )
–
Appendix
Table 2: Full results in % with mean (standard error) over three seeds.
Group-Mean
Leave-One-Out
Method
Seed 1
Seed 2
Seed 3
Mean
Seed 1
Seed 2
Seed 3
Mean
Sudoku
Single
75.0
73.1
74.4
74.2 ( 0.6 )
72.5
75.3
72.4
73.4 ( 1.0 )
Stratified
80.0
80.0
80.8
80.3 ( 0.3 )
74.2
76.1
77.5
75.9 ( 1.0 )
Gap
+5.0
+6.9
+6.4
+6.1 ( 0.6 )
+1.7
+0.8
+5.1
+2.5 ( 1.3 )
K&K
Single
91.2
91.2
89.4
90.6 ( 0.6 )
88.9
84.8
85.9
86.5 ( 1.2 )
Stratified
94.7
92.7
92.1
93.2 ( 0.8 )
91.9
85.0
88.5
88.5 ( 2.0 )
Appendix
Table 3: Comparison between group-mean (Eq. 6 ) and leave-one-out (Eq. 4 ) advantage.
Figure 9: Cumulative number of updates per adapter during training for the group-mean (Eq. 6 ) and LOO (Eq. 4 ) advantages. For each task, the panel on the right shows the final update counts for adapters ranked by final counts. Group-mean provides a denser learning signal, which leads to more updates for every adapter, and the updates are also spread relatively evenly across adapters.
Figure 10: Pass@ 5 accuracy of Mixture versus Stratified under the group-mean (Eq. 6 ) and LOO (Eq. 4 ) advantages on Knights-&-Knaves and Sudoku tasks. Error bars denote standard error across three seeds. Each panel’s y -axis is truncated (kink mark) to make the gap visible. There is a substantial gap between Stratified and Mixture on both tasks under both estimators.
Individual Adapters in Stratified Population
Method
Seed
Rank 1
Rank 2
Rank 3
Rank 4
Rank 5
Single
Stratified
Sudoku
1
46.2
42.6
41.8
39.6
37.7
75.0
80.0
2
45.7
42.0
40.9
40.6
40.6
73.1
80.0
3
48.1
44.6
40.8
40.3
38.0
74.4
80.8
K&K
1
69.1
57.1
54.8
51.5
39.4
91.2
94.7
2
67.0
65.2
61.9
60.9
35.0
91.2
92.7
Appendix
Table 4: Pass@ 5 accuracy of individual LoRA adapters in our stratified population of five specialized policies, compared to Single and Stratified , with adapters ranked by their pass@ 5 accuracy. Even the best LoRA adapter in the population underperforms both Single and Stratified .
Subset Size
Best Coverage
Gap
2
74.6%
19.0
3
84.6%
9.0
4
90.8%
2.8
5
93.6%
0.0
Appendix
Table 5: Coverage of the best subset of learned LoRA adapters on Knights-&-Knaves using greedy decoding. The gap is w.r.t. the full population.
Figure 11: Pass@ k accuracy (averaged across three seeds) on Sudoku for different population sizes k , with results for individual seeds shown as faint dots. Stratified consistently beats Single .