Extending CoT through RL has been widely used to enhance the reasoning capabilities of LLMs. However, due to the sparsity of reward signals, it can also induce undesirable thinking patterns such as overthinking, i.e., generating redundant intermediate reasoning content. In this work, we argue that a major source of such redundancy is inefficient reflection, which often manifests in two problematic patterns: Indiscriminate Reflection, where the model performs broad, low-impact checks throughout reasoning, and Repetitive Reflection, where it repeatedly re-verifies an already established conclusion. To address this, we introduce a graph-based CoT optimization framework. Specifically, we convert each linear CoT into a directed acyclic graph (DAG) with explicit dependency edges, and design a dual pruning strategy: branch-level pruning removes weakly contributing reflection branches, while depth-level pruning eliminates late-stage re-verification. We distill this behavior via a three-stage pipeline: (1) SFT to initialize the policy on pruned concise traces, (2) DPO to prefer correct but less redundant trajectories, and (3) GRPO with length penalty to jointly optimize answer correctness and efficiency. Experiments show that our approach reduces the average reasoning tokens by 42% while maintaining or improving accuracy.
Figures & tables
Figure 1: Relationship between average reasoning tokens and accuracy averaged across all benchmarks for different methods.
Figure 2: Training pipeline of our graph-based redundancy-aware post-training framework. We first perform cold-start SFT using compressed chain-of-thoughts obtained via iterative graph construction and redundancy pruning, which instills an inductive bias toward concise reasoning. On top of the SFT model, we conduct DPO by ranking correct trajectories according to redundancy, preferring low-redundancy outputs over highly redundant ones. Finally, we apply GRPO with length penalty, jointly optimizing answer correctness and reasoning efficiency to obtain the final model.
Method
AIME24
AIME25
AMC23
OlympiadBench
MATH500
Average
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Acc ↑
Len ↓
Open-Source R1-Style Models
Skywork-OR1-7B He et al. (2025)
37.08
11890
24.67
12831
73.50
5241
51.41
6075
85.96
3753
54.52
7958
OREAL-7B Lyu et al. (2025)
38.75
10907
28.75
11309
81.25
4510
56.44
5164
88.83
3023
58.80
6983
AReaL-boba-RL-7B Mei et al. (2025)
36.67
11407
25.42
12365
71.56
5337
52.91
6037
87.12
3904
54.74
7810
Light-R1-DS-7B Wen et al. (2025a)
39.67
12036
31.00
12498
78.25
6073
54.74
5306
87.58
3020
58.25
7787
Table 1: Performance comparison across multiple math reasoning benchmarks (Accuracy ↑ and Average Length ↓ ). ∗ We adapt the original implementation of O1-Pruner and TokenSkip to the DeepSeek-R1-Distill-Qwen models.
Figure 3: Stage-wise ablation across five benchmarks. (a) Accuracy (%). (b) Relative reasoning length measured by the average number of generated tokens, normalized to the Base setting (token ratio; Base =1.0, lower is better). The compared configurations follow a cumulative training recipe: starting from Base , we sequentially add SFT , then DPO , and finally GRPO .
Statistic
Full CoT
Pruned CoT
Total Samples
3335
3335
Avg. Nodes
27.8
15.6
Avg. Review Nodes
16.8
4.5
Avg. Tokens
6468
4439
Total Cost
–
$20
Table 2: Dataset and graph statistics before and after pruning.
Figure 4: Changes in reasoning length and reasoning token usage before and after training on AIME24.
CoT Variant
Accuracy ( ↑ )
Consistency ( ↑ )
Full-CoT
98.95
99.60
Graph-Pruned
93.70
90.69
Len-Trunc
73.60
69.10
Table 3: Comparison of accuracy and consistency across different Chain-of-Thought (CoT) pruning methods.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Node Type
Precision
Recall
F1
Review
0.9048
0.9661
0.9344
Progress
0.8947
0.8500
0.8718
Atomicity Valid (%)
85.29
Appendix
Table 4: Type classification performance and step atomicity validity of constructed graph nodes.
Name
Value
Epochs
8
Global batch size
64
Max sequence length
8192
Optimizer
AdamW
Learning rate
1×10−5
LoRA rank
32
Appendix
Table 5: Hyper-parameters for SFT training.
Name
Value
Epochs
5
Global batch size
64
Max sequence length
8192
Optimizer
AdamW
Learning rate
1×10−7
Appendix
Table 6: Hyper-parameters for DPO training.
Name
Value
KL coefficient β
1×10−3
Rollouts per input
8
Sampling temperature
1.0
Max response length
12000
Global batch size
64
Optimizer
AdamW
Appendix
Table 7: Hyper-parameters for GRPO training.
Figure 5: RL training dynamics for different model scales. (a) 7B; (b) 1.5B. Left: mean reward. Right: response length (tokens).
Figure 6: Instruction for updating a graph-structured chain-of-thought
Figure 7: Qualitative comparison of reflection behavior between the base model and our trained model. The left column shows the original CoT, and the right column shows its graph-structured representation. Red-highlighted text indicate reflection-related content. Compared to the base model, our trained model exhibits reduced and more focused reflection.