Self-refinement, typically structured as generation, critique, and revision, is a widely adopted paradigm for improving LLM generation and serves as a core mechanism in many LLM agents. While the three stages involve different cognitive demands, most existing approaches conveniently treat the model size as an implementation detail rather than a subject of study, which may lead to a waste of resources. Little work has systematically examined how model size affects each stage or whether effective self-refinement requires equally capable models for generation, critique, and revision. We present the first stage-wise model size study of the self-refinement pipeline on 5 benchmarks from different domains using 6 model sizes of Qwen3 and 4 model sizes of Gemma 3. We conclude that larger generators and refiners generally improve the pipeline, whereas an undersized refiner can even harm performance. Second, performance is highly insensitive to the size of the critic, although including even a small critic consistently outperforms omitting critique altogether. Our findings demonstrate that model capacity should not be allocated uniformly across self-refinement pipelines. Instead, different stages exhibit distinct size scaling characteristics, providing practical guidance for designing more computationally efficient multi-stage language model systems.
Figures & tables
Figure 1: Overview of our three-stage self-refinement pipeline with a simplified example. Given an input x , the generator produces an initial solution y0 , the critic provides natural-language feedback f on x and y0 , and the refiner revises the solution based on x , y0 , and f .
Figure 2: Stage-wise model size analysis. The three panels illustrate the generator, critic, and refiner sweeps. The highlighted stage is the stage under evaluation, whose model size is varied while the other two stages are held fixed. We also compare against a no-critique baseline.
Figure 3: Representative examples from the five benchmarks. The first column briefly illustrates the objective and input for each task, followed by the initial generation, the generated critique, and the final refined solution, demonstrating how the self-refinement pipeline operates across diverse domains.
Figure 4: Stage-wise model-size sweeps across five benchmarks under the Qwen3-32B configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2 .
Benchmark
Model
Generator
Critic
Refiner
Δ
Std.
Δ
Std.
Δ
Std.
Meeting Planning
Qwen
16.00
6.15
9.00
3.10
32.00
11.20
Gemma
24.00
8.76
8.00
2.92
34.00
12.64
CNN/DailyMail
Qwen
10.00
3.95
8.00
3.10
23.40
8.99
Gemma
25.10
9.62
6.30
2.24
16.80
6.39
ZebraLogic
Qwen
8.65
3.07
3.89
1.46
13.39
4.71
Table 1: Performance variability across model sizes when varying one stage while fixing the other two at the largest model. Values are reported in each benchmark’s evaluation metric, as specified in Section 4.2 .
Figure 5: Distribution of manual critique-quality scores across five benchmarks. For each benchmark (top to bottom: Meeting Planning, CNN/DailyMail, ZebraLogic, PIE, and CollaboSentGen): Stacked segments show the percentage of critiques assigned scores 1–5, while the percentages on the right report the fraction of non-misleading critiques (scores 3–5).
Input
Start: The Castro (9:00 AM) Ronald: Chinatown (9:30–11:30, 75 min) Nancy: Nob Hill (11:15–12:00, 30 min)
Generated
09:30–10:30 Ronald 10:50–11:20 Nancy
0.6B Critic (Score 4)
Detects: Ronald’s meeting is only 60 min (requires 75 min).
32B Critic (Score 5)
Detects: Ronald’s meeting is only 60 min. Detects: Nancy is scheduled outside her availability window.
Table 2: Representative example illustrating performance of 0.6B critic and 32B critic.
Benchmark
Generator Sweep
Refiner Sweep
Meeting Planning
+9.67
+10.34
CNN/DailyMail
+4.56
+5.21
ZebraLogic
+7.97
+7.86
PIE
+9.03
+11.06
CollaboSentGen
+3.71
+4.01
Table 3: Average critique gain with Qwen3-0.6B as the critic over the matched no-critique baseline across the generator and refiner sweeps.
Figure 6: A degradation example from ZebraLogic. Despite a correct critique, the 0.6B refiner is introducing new errors while the 32B refiner is incorporating the critique.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Stage-wise model-size sweeps across five benchmarks under the Gemma 3-27 configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep respectively. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2
Figure 8: Stage-wise model-size sweeps across five benchmarks under the Qwen3-14B configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep respectively. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2
Figure 9: Stage-wise model-size sweeps across five benchmarks under the Qwen3-8B configuration. X-axis is the model sizes. Blue line, green line, and the red line represent Generator sweep, Critic sweep and the Refiner sweep respectively. The grey dashed line reports the corresponding initial-generation performance without refinement. Steeper curves indicate greater sensitivity to the model size of that stage. Metrics are defined in Section 4.2
Table 4: Refiner sizes that perform below the corresponding initial generation under different model configurations. A checkmark indicates that using the corresponding refiner size results in lower performance than the initial generation.
Reflective prompt optimization improves large language model (LLM) systems without updating model weights, but fixed architectures constrain how self-refinement is organized. We introduce Workflow-Designing Agents (WDA), a framework for streamlined reflective evolution of task-adaptive self-refinement pipelines. Starting from a minimal prompt, WDA jointly evolves stage instructions and their sequential structure. During evolution, we find that repeated revisions can accumulate redundant instructions in a single prompt. In WDA, we propose to address this problem with SPLIT, which redistributes these instructions across specialized stages. Three-example reflection and local screening guide selective search, while calibration scores guide Pareto admission and rollback of unhelpful trailing updates. The resulting pipelines are task-adaptive: their instructions and depth are learned from task data, then fixed for all test inputs within that task. We evaluate WDA on five benchmarks spanning knowledge, mathematical reasoning, multi-hop question answering, and instruction following. On Qwen3.5-9B, WDA achieves an average score of 51.24%, improving over the initial solver by 8.63 percentage points and the variant without SPLIT by 2.60 points. On GPT-4.1-mini, it achieves 49.00%, with corresponding gains of 5.62 and 3.69 points. These results support task-adaptive self-refinement as a complementary direction to broader agentic workflow search.
We study self-modeling: an LLM's ability to answer questions about its own behavior. We focus on verifiable behavioral questions, such as whether a prompt edit would change the model's final answer. To measure this capability, we introduce a benchmark that tests diverse types of self-modeling questions. Current models show non-trivial but limited self-modeling skill, and make systematic mistakes on simple counterfactual questions about their own behavior. To improve self-modeling skill, we develop a scalable synthetic-data pipeline that produces self-modeling training data, and show that reinforcement-learning can improve aggregate self-modeling skill across three open-source model families with some transfer to held-out tasks. These gains, however, do not seem to constitute introspection consistently: improved self-modeling may not arise from privileged access to the model's internal decision process.
Siqi Zeng, Andre N. Assis, Rowan Wang
University of Illinois · Constellation · Anthropic
We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.
Difan Jiao, Qianfeng Wen, Blair Yang +2
Department of Computer Science, University of Toronto · Coolwei AI Lab