The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis

Jinyang Zhang; Weibin Liao; Keqin Bao; Sihang Li; Shaobo Wang; Muyang Ye; Hongxin Ding; Yue Fang

cs.CLSep 13, 2026

The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis

Authors: Jinyang Zhang, Weibin Liao, Keqin Bao, Sihang Li, Shaobo Wang, Muyang Ye, Hongxin Ding, Yue Fang,

Organizations: School of Computer Science, Peking University · Qwen Team, Alibaba Group · EPIC Lab, SJTU · College of Computer Science and Technology, Zhejiang University.

Abstract

Large Language Models (LLMs) excel at programming tasks but frequently fail at deterministic, fine-grained reasoning in natural language, relying heavily on semantic approximations rather than robust symbolic execution. To bridge this gap, we propose MIMIC, a framework that leverages executable code as a rigorous medium for reasoning data synthesis. MIMIC fundamentally transforms algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation. Crucially, these explicit intermediate execution states naturally form a Code-Instrumented Reward (CIR), providing dense, high-fidelity process supervision for reinforcement learning without external reward models. Extensive evaluations reveal that models trained via SFT and GRPO on our synthesized dataset achieve substantial, consistent gains. Our method significantly elevates accuracy across general reasoning, complex mathematical benchmarks, and fine-grained deterministic tasks, demonstrating that the procedural rigor of executable code can effectively unlock and enhance the generalized reasoning capabilities of LLMs. Our code and data are available at https://github.com/zjy1298/MIMIC.

Figures & tables

Figure 1: The paradox of LLMs: while they excel at complex programming logic, they often fail at basic deterministic natural language reasoning. MIMIC bridges this gap by leveraging executable code to generate hallucination-mitigated reasoning trajectories.

Figure 2: Overview of the MIMIC framework. It transforms raw algorithmic problems into verifiable reasoning data through a four-stage pipeline: (1) Problem Classification and Mathematical Kernel Extraction, (2) Problem Synthesis via Narrative Fusion, (3) Code-Guided Test Synthesis and Narrative Input Fusion, and (4) Trace-Grounded Chain-of-Thought (CoT) Synthesis.

Model	Setting	General Reasoning							Math					Code
Model	Setting	ARC	BBH	MIMIC	GPQA	MMLU	DROP	Avg	GSM8K	GSM+	MATH	AIME	Avg	HEval	CruxE	MBPP	Avg
(A) Main Results: Baseline $\rightarrow$ Ours (SFT) / Ours (GRPO)
Qwen3-4B	Base	79.6	76.0	40.6	40.1	66.3	50.5	58.9	89.8	73.2	78.3	15.6	64.2	31.7	69.3	39.7	46.9
	Ours (SFT)	85.9	77.0	43.5	40.4	67.3	55.5	61.6 $\uparrow$ 2.7	91.0	74.9	77.6	16.7	65.0 $\uparrow$ 0.8	32.1	70.8	38.8	47.2 $\uparrow$ 0.3
	Ours (GRPO)	80.6	77.9	51.3	40.8	67.5	58.3	62.7 $\uparrow$ 3.8	91.0	76.4	80.3	28.3	69.0 $\uparrow$ 4.8	31.7	72.7	43.6	49.3 $\uparrow$ 2.4
Qwen3-8B	Base	79.8	78.8	41.2	44.4	70.5	46.6	60.2	90.6	73.7	79.2	23.3	66.7	22.6	70.6	40.5	44.6

Table 1: Main Results and Ablation Studies. (A) Main results comparing base models with our MIMIC data via SFT and GRPO. (B) Data source ablation: our contextualized synthesis vs. raw Codeforces problems. (C) Cross-domain mixing with mathematical/benchmark data. (D) RL algorithm robustness. Avg = category mean; subscripts show absolute change vs. the reference row. Best per group in bold .

Model	Setting	General Reasoning							Math					Code
Model	Setting	ARC	BBH	MIMIC	GPQA	MMLU	DROP	Avg	GSM8K	GSM+	MATH	AIME	Avg	HEval	CruxE	MBPP	Avg
Comparison with Closely Related Baselines
(A) SFT
Qwen3-4B	Ours (SFT)	85.9	77.0	43.5	40.4	67.3	55.5	61.6	91.0	74.9	77.6	16.7	65.0	32.1	70.8	35.8	46.2
	CoE (SFT)	85.2	43.9	28.9	25.4	44.8	42.6	45.1 $\downarrow$ 16.5	71.5	54.9	40.4	6.7	43.4 $\downarrow$ 21.6	34.2	65.6	34.6	44.8 $\downarrow$ 1.4
	ExecGrounded (SFT)	75.4	66.9	39.2	35.0	40.8	26.1	47.2 $\downarrow$ 14.4	77.6	63.4	74.4	16.7	58.0 $\downarrow$ 7.0	23.8	63.0	45.5	44.1 $\downarrow$ 2.1

Table 2: Comparison with closely related baselines. We compare MIMIC against representative recent methods under both SFT and GRPO. Avg = category mean; subscripts show absolute change vs. the reference row. Best per group in bold .

Model	Method	Maze	String	Char	Char+CoT
Qwen3-4B	Base	16.0	13.6	37.9	97.1
	+SFT	18.0	22.8	40.4	97.8
	+GRPO	22.0	26.8	39.1	98.2
Qwen3-8B	Base	22.0	26.2	38.6	98.0
	+SFT	22.0	37.5	39.3	98.6
	+GRPO	40.0	39.5	41.3	99.2

Table 3: Character-level and spatial reasoning benchmarks. We evaluate on benchmarks targeting fine-grained perception and rules (e.g., comparing string patterns like 9.11 and 9.9): CharBench, StringBench, and MazeBench. Best per model family in bold .

Format	Acc. (%)	Code Rate (%)	Acc. (%) w/ Code	Acc. (%) w/o Code
Original	23.8	38.7	18.4	27.2
Refined	34.4	29.9	30.6	36.0
Fused (Ours)	42.2	0.7	15.0	42.4
Synthesized	61.8	0.8	41.3	62.0

Table 4: Preliminary comparison of different problem presentation formats on Qwen3-8B . “Code Rate” denotes the proportion of cases where the model attempted to write code during reasoning.

Model	Reward	General Reasoning							Math					Code
Model	Reward	ARC	BBH	MIMIC	GPQA	MMLU	DROP	Avg	GSM8K	GSM+	MATH	AIME	Avg	HEval	CruxE	MBPP	Avg
Qwen3-8B	Base	79.8	78.8	41.2	44.4	70.5	46.6	60.2	90.6	73.7	79.2	23.3	66.7	22.6	70.6	40.5	44.6
	Binary (GRPO)	81.2	81.7	52.2	42.6	72.2	53.1	63.8 $\uparrow$ 3.6	91.7	78.1	81.0	26.7	69.4 $\uparrow$ 2.7	23.8	76.3	44.0	48.0 $\uparrow$ 3.4
	CIR (Ours)	82.7	80.6	54.5	48.5	71.6	53.5	65.2 $\uparrow$ 5.0	92.6	78.0	81.4	30.0	70.5 $\uparrow$ 3.8	25.3	76.2	40.9	47.5 $\uparrow$ 2.9
	PRM800K	79.8	79.3	43.8	42.7	71.6	50.9	61.4 $\uparrow$ 1.2	91.1	74.7	80.5	23.3	67.4 $\uparrow$ 0.7	23.2	71.8	40.5	45.2 $\uparrow$ 0.6
	Math-Shepherd	81.4	79.8	43.9	43.1	71.4	52.7	62.1 $\uparrow$ 1.9	90.9	74.3	75.9	20.0	65.3 $\downarrow$ 1.4	21.3	72.3	44.4	46.0 $\uparrow$ 1.4

Table 5: Reward Signal Ablation. Comparing reward strategies under GRPO training: (1) Binary outcome reward; (2) CIR (Code-Instrumented Reward), our approach leveraging execution checkpoints; (3) PRM800K ; (4) Math-Shepherd . Subscripts show absolute change vs. Base.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Category	Subtype	Description
Type 1	Text-Encoded Visual / Spatial Reasoning	1.1 Grid / board / pixel encoding	Layouts represented as character matrices or ASCII art, where the solver must reconstruct a 2D structure from text and reason about adjacency, connectivity, or shape.
		1.2 Geometry / coordinates / directions	Spatial reasoning grounded in language, such as left/right/up/down relations, segment intersection, coordinate movement, or point-in-polygon logic.
		1.3 Glyph or compositional character structure	Reasoning based on visual properties of symbols or characters, including radicals, stroke counts, mirrored forms, rotated shapes, or internal composition.
		1.4 Textual charts / tabular layout understanding	Information encoded through alignment, indentation, spacing, or fixed-width formatting, such as tree structures or visually organized tables.
Type 2	Sub-Token / Character Manipulation	2.1 Character-level counting / extraction / replacement	Fine-grained operations on strings, including counting symbol frequency, extracting characters at specified positions, or performing local edits and substitutions.
		2.2 Character-wise mapping and ciphers	Per-character transformation rules such as Caesar shifts, ASCII-based conversion, or Base64-style encoding and decoding.

Appendix

Table 6: Reasoning taxonomy used in Stage 1. Each problem is assigned to the category that best reflects its dominant reasoning bottleneck.

Benchmark	HF Dataset	Dataset Summary	Primary Abilities Tested
Mathematical Reasoning
GSM8K	gsm8k	Grade-school math word problems (1,319 test) requiring 2–8 steps of elementary arithmetic Cobbe et al. (2021) .	Multi-step arithmetic reasoning; text-to-equation mapping.
GSM-Plus	GSM-Plus	Adversarially augmented GSM8K Li et al. (2024) with numerical perturbation, structural rephrasing, and distractors to reduce shortcut biases.	Robust numerical reasoning under perturbation.
MATH	competition_math	5,000 competition-level problems across 7 subjects (algebra, geometry, number theory, etc.) drawn from AMC/AIME contests Hendrycks et al. (2021) .	Formal math reasoning; symbolic manipulation; multi-step proofs.
AIME	AIME_2024 AIME_2025	60 problems from AIME 2024 and AIME2025 (I+II) Zhang et al. (2023) ; integer answers in 0–999 requiring creative multi-step reasoning in algebra, combinatorics, and geometry.	Competition-level mathematical problem solving.
General Reasoning & Knowledge

Appendix

Table 7: Benchmarks used in our evaluation, grouped into four categories.

Hyperparameter	Value
Fine-tuning type	Full parameter
Parallelism	DeepSpeed ZeRO-3
Number of GPUs	8
Per-device batch size	2
Gradient accumulation steps	8
Effective batch size	128

Appendix

Table 8: SFT training hyperparameters.

Hyperparameter	Value
Algorithm	GRPO
Parallelism	FSDP
Number of GPUs	8
Rollout engine	vLLM
Rollout samples per prompt ( $n$ )	8
Train batch size	128

Appendix

Table 9: GRPO training hyperparameters.

Figure 3: Hierarchical distribution of Codeforces problems in the Stage-I taxonomy. The inner ring denotes the five top-level categories and the outer ring denotes the corresponding fine-grained subtypes. Colors indicate top-level categories, while lighter shades represent their subtypes. Sector sizes are normalized within each level for visual clarity, and the actual numbers of instances are shown inside the segments.

Top-level Category	Subtype (abbrev.)	Count	Top-level Share (%)	Subtype Share (Parent, %)
Type 1: Text-Encoded Visual/Spatial	1.1 Grid/ASCII Parsing	135	2.07	65.2
	1.2 Geom./Rel. Grounding	69		33.3
	1.4 Layout-aware Tables/Diag.	2		1.0
	1.3 Glyph Decomp.	1		0.5
Type 2: Sub-Token/ Character	2.1 Char Count/Extract/Replace	527	6.47	81.3
	2.2 Char Mapping/Crypto.	61		9.4

Appendix

Table 10: Hierarchical distribution of the cleaned Stage-I dataset. “Top-level Share” is computed over all 10,008 instances, and “Subtype Share (Parent)” denotes the proportion within each top-level category. Abbreviated subtype names are used for space efficiency.

Rating	Original	Refined	Fused	Synthesized
$<1200$	27.7	20.2	0.4	0.6
1200–1599	46.3	34.9	0.6	0.8
1600–1999	49.2	38.1	0.8	0.9
$\geq 2000$	44.9	35.5	0.8	0.7

Appendix

Table 11: Code-writing rate in model reasoning across different Codeforces rating bands under different problem presentation formats. Lower is better.

Rating	Original	Refined	Fused	Synthesized	$\Delta$
$<1200$	46.6	55.1	48.6	72.3	$-6.6$
1200–1599	32.2	44.4	45.8	68.6	$+1.4$
1600–1999	19.3	30.6	41.4	65.1	$+10.8$
$\geq 2000$	13.0	24.4	40.6	56.9	$+16.2$

Appendix

Table 12: Accuracy (%) across different Codeforces rating bands under different problem presentation formats. The last column

\Delta

reports the accuracy gain of Fused over Refined .

Type	Accuracy (%)	Total	Code Rate (%)
Visual/Spatial	37.2	2328	0.5
Char-Level	47.2	8045	1.2
Math	44.7	54121	0.9
Data Process	53.5	6011	0.5
Rule/State	37.2	57090	0.6

Appendix

Table 13: Model performance across top-level taxonomy types under the fused setting.

Rating	Official	Level A	Level B	Level C
$<1200$	52.4	51.8	48.3	42.8
1200–1599	44.8	51.9	48.0	38.4
1600–1999	36.1	50.7	45.1	33.1
2000–2399	31.5	52.0	46.0	32.2
$\geq 2400$	24.1	48.4	42.2	28.0

Appendix

Table 14: Accuracy (%) across different test sources and Codeforces rating bands.

Method	Unshifted	Marginal	Shifted
SFT vs Base	78.9%	15.6%	5.5%
GRPO vs Base	95.1%	4.8%	0.2%

Appendix

Table 15: Shifted token distribution under different alignment methods, measured against the base model. We report the proportion of unshifted, marginal, and shifted positions.

Figure 4: Word cloud of shifted tokens under SFT, measured relative to the base model. The shifted vocabulary is dominated by instruction- and description-level terms such as requested , allowed , sequence , algorithm , and grid , suggesting that SFT learns substantial surface-level regularities of the training data in addition to task behavior.

Figure 5: Word cloud of shifted tokens under GRPO, measured relative to the base model. Although the number of shifted positions is much smaller, the shifted vocabulary is more concentrated on reasoning-relevant terms such as count , sum , configuration , operation , and determined , indicating more targeted changes to computation and decision-making behavior.

Panel of Figure 6 — Figure 6: GRPO training dynamics for Qwen3-8B on $\mathcal{D}_{\text{mixed}}$ . (a) Mean reward increases steadily from 0.61 to 0.81, indicating consistent learning. (b) KL divergence remains bounded below 0.004 (smoothed), confirming stable policy updates without reward hacking. (c) Mean response length grows from $\sim$ 850 to $\sim$ 1,330 tokens, reflecting the emergence of more thorough reasoning chains.

Figure 7: Benchmark scaling during GRPO training (Qwen3-8B,

\mathcal{D}_{\text{mixed}}

). All four benchmarks show consistent improvement over 190 training steps. Dashed lines indicate base model performance. MIMIC improves by +9.8%, demonstrating effective transfer from code-mediated reasoning to general problem-solving. Notably, MBPP (+8.9%) also improves substantially, suggesting that reasoning-focused training can reciprocally enhance code generation capabilities.

Figure 8: SFT training and validation loss for Qwen3-8B on

\mathcal{D}_{\text{mixed}}

. The training loss drops sharply from 0.92 to

\sim

0.45 within the first 400 steps, then gradually converges to 0.39. The validation loss closely follows, plateauing at 0.39 with minimal train-validation gap.

Figure 9: Benchmark evolution during SFT (Qwen3-8B,

\mathcal{D}_{\text{mixed}}

). Downstream benchmarks show quick initial gains that plateau or slightly decline after step 800, contrasting sharply with the continuous improvement observed under GRPO (Figure 7 ).

Model	Setting	General Reasoning							Math					Code
Model	Setting	ARC	BBH	MIMIC	GPQA	MMLU	DROP	Avg	GSM8K	GSM+	MATH	AIME	Avg	HEval	CruxE	MBPP	Avg
Qwen3-8B	Base	79.8	78.8	41.2	44.4	70.5	46.6	60.2	90.6	73.7	79.2	23.3	66.7	22.6	70.6	40.5	44.6
Qwen3-8B	Ours (All)	81.2	81.7	52.2	42.6	72.2	53.1	63.8 $\uparrow$ 3.6	91.7	78.1	81.0	26.7	69.4 $\uparrow$ 2.7	23.8	76.3	44.0	48.0 $\uparrow$ 3.4
(A) Single-Type Training: GRPO with only one task category
Qwen3-8B	Type 1 (Visual)	79.3	78.8	43.0	45.2	71.1	50.8	61.4 $\uparrow$ 1.2	90.3	74.6	79.9	26.7	67.9 $\uparrow$ 1.2	22.9	71.4	41.6	45.3 $\uparrow$ 0.7
	Type 2 (Char)	80.0	79.7	42.9	46.1	71.4	51.5	61.9 $\uparrow$ 1.7	92.0	75.4	81.1	28.3	69.2 $\uparrow$ 2.5	22.9	70.4	43.6	45.6 $\uparrow$ 1.0

Appendix

Table 16: Task Type Ablation (Qwen3-8B). (A) Single-type training : GRPO trained with only one task category. (B) Leave-one-out : GRPO trained with all categories except one. Subscripts on Avg show absolute change vs. Base. Bold = best, underline = second best within each section.

Figure 10: LOO Impact on category averages (Reasoning, Math, Code). Positive values (green) indicate beneficial contribution; negative values (red) indicate interference when included.

Figure 11: Per-benchmark LOO Impact. Each cell shows the accuracy drop (in percentage points) when the corresponding task type is removed from training. Horizontal lines separate the three evaluation categories: Reasoning, Math, and Code.

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 293 papers

Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning
May 18, 2026Zhanyue Qin, Jia Feng, Yibo Lyu +4Reinforcement Learning RL for Language Model Reasoning
Harbin Institute of Technology · The Chinese University of Hong Kong
Learning from Synthetic Data Improves Multi-hop Reasoning
Mar 2, 2026Anmol Kabra, Yilun Yin, Albert Gong +6RL for Language Model Reasoning Multi-Hop Reasoning
Cornell University · University of Cambridge · Stanford University
Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning
Apr 26, 2026Zichuan Fu, Xian Wu, Guojing Li +7Efficient Language Model Reasoning LLM Inference Acceleration
City University of Hong Kong · Tencent Jarvis Lab · Renmin University of China +1

May 18, 2026J/K move · Enter open · S save

Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning

Zhanyue Qin, Jia Feng, Yibo Lyu, Yun Peng+3

Harbin Institute of Technology · The Chinese University of Hong Kong

Reinforcement Learning RL for Language Model Reasoning

Open paper AskPDF arXiv

Code reasoning refers to the task of predicting the output of a program given its source code and specific inputs. It can measure the reasoning capability of large language models (LLMs) and also benefit downstream tasks such as code generation and mathematical reasoning. Existing work has verified the effectiveness of reinforcement learning on the task. However, these methods design rewards solely based on final outputs or coarse-grained signals, and neglect the inherent consistency of the stepwise reasoning process in the task. Therefore, these methods often result in sparse reward or reward hacking, which limits the full play of enhanced learning capabilities. To alleviate these issues, we propose CodeThinker, a consistency-driven reinforcement learning framework for code reasoning. Specifically, CodeThinker has three key components: (1) a stepwise reasoning-aware model training module, which utilizes a consistency tracing paradigm as a template to synthesize training data that captures the stepwise reasoning process; (2) a dynamic beam sampling strategy, which aims to improve the quality of sampled outputs under a fixed sampling budget; and (3) a consistency reward mechanism that can effectively alleviate reward hacking. Experiments on three popular benchmarks show that CodeThinker achieves state-of-the-art performance across multiple LLMs. For instance, it outperforms the strongest baseline by 4.3% in accuracy when deployed on Qwen2.5-Coder-7B-Instruct. We also validate the effectiveness of CodeThinker on downstream tasks. Results show that, without additional training, CodeThinker obtains average accuracy gains of 5.33 and 3.11 percentage points on mathematical reasoning and code reasoning tasks covering 17 programming languages, respectively.

Abstract

Synthetic Data Generation for Language Models

Explore similar work

Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning

Learning from Synthetic Data Improves Multi-hop Reasoning

Tandem: Riding Together with Large and Small Language Models for Efficient Reasoning