Recent large language models achieve strong performance on complex reasoning tasks, where reinforcement learning with Group Relative Policy Optimization (GRPO) has emerged as a leading paradigm for optimizing models on self-generated trajectories. However, the on-policy nature of GRPO bounds the model to the reasoning skills it can already produce, restricting to learn more advanced capabilities. Prior works inject privileged reasoning traces from a stronger teacher policy to guide training, yet these traces are inherently out of distribution with respect to the student policy. We observe that this mismatch between on-policy and off-policy causes gradient clipping on semantically critical reasoning tokens, ultimately rewarding correct answers while leaving the reasoning that justifies them unlearned. Hence, we propose Echo-GRPO, a framework that lets the model reason in the words it speaks. Rather than imitating low-probability privileged traces from the teacher model, Echo-GRPO rewrites them into the student policy's own idiolect, that is, its own characteristic vocabulary and expression patterns, while preserving their semantics via Dual-Reference Decoding. We instantiate this framework as VideoEcho-R1 for video reasoning distillation, achieving consistent improvements across three multimodal LLM backbones and five benchmarks. Finally, we show that our idiolectal paraphrasing is a plug-in module that consistently improves both RL and supervised fine-tuning frameworks for reasoning distillation, demonstrating that policy-aligned supervision extends beyond GRPO.
Figures & tables
Figure 1: Reasoning in the words you know. Token-level log-probabilities under the policy πθ (red shading indicates low probability). Top: Mixed-Policy GRPO forces the policy to imitate privileged reasoning traces from a stronger teacher, leaving semantically critical ones, such as “school of fish,” in low-likelihood regions. Bottom: Echo-GRPO paraphrases the same reasoning into the policy’s native distribution (“school of fish” → “several fish”), preserving the semantics while keeping all tokens within the policy’s distribution.
Figure 2: (a) (top) Number of samples containing at least one clipped token throughout training for Mixed-Policy GRPO (red), and Echo-GRPO (blue). (bottom) Ratio of semantically important tokens ( e.g. , nouns from the question and options) among all clipped tokens for each method. (b) Token-level visualization of clipping behavior on a representative video reasoning sample, where grey boxes indicate clipped tokens and red shading denotes per-token loss intensity.
Clipping Method
Priv. Trace
Avg.
Δ
Mixed-Policy
OOD
35.9
–
+ Unclip all tokens
OOD
40.8
+4.9
+ Unclip random tokens
OOD
48.4
+12.5
+ Unclip semantic tokens
OOD
52.0
+16.1
Echo-GRPO (Ours)
ID
58.2
+22.3
Table 1: Effect of clipping on Mixed-Policy failure. OOD and ID refer to Out-of-distribution and In-distribution, respectively. Performance is averaged over three major benchmarks: VideoMMMU, MMVU, and Video-Holmes.
Figure 3: Overview of Echo-GRPO. Given a video question and a privileged reasoning trace y~ , Echo-GRPO adopts an idiolectal paraphrased trace yrewrite produced by Dual-Reference Decoding (DRD). DRD combines two conditional distributions of πθ0 as a product of distributions: the semantic reference (yellow) constrains token selection to be faithful to y~ , correcting the base policy’s preference for semantically incorrect tokens ( e.g. , ‘bag’ to ‘box’); while distribution alignment (blue) resolves ties among semantically equivalent candidates by preferring tokens the student policy is more likely to generate ( e.g. , ‘visible’ over ‘shown’), keeping the rewritten trace aligned to its naive distribution.
Figure 4: Prompt for idiolectal paraphrasing. The privileged reasoning trace y~ corresponds to ( {{PRIVILEGED REASONING TRACE}} ). We append the question and options at the end of the prompt. For the Question and Options we follow the format of OneThinker [ 21 ] . Further details are in the supplement.
Figure 5: Training Dynamics of Echo-GRPO.
Figure 6: Training Dynamics of idiolectal paraphrasing as a plug-in.
Figure 8
Method
Priv.
V-MMMU
MMVU
VHolm
Avg.
SFT
y~
48.2
62.9
40.8
50.6
yrewrite
56.3
64.0
37.6
52.6
SFT → GRPO
y~
57.5
66.2
44.0
55.9
yrewrite
58.2
68.8
49.5
58.8
Table 5: Comparison of SFT and SFT → GRPO using offline ( y~ ) and idiolectal paraphrased traces ( yrewrite ).
Figure 9: Qualitative example. Mixed-Policy GRPO (top) mimics the structural format of privileged traces, defaulting to narrative speculation (“this is part of a fictional series, so the answer is likely…”) and predicting the wrong answer (A: Neighbor). Echo-GRPO (bottom) instead references concrete events from the video (highlighted in green) and reaches the correct answer (B: Kidnapper).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
top- k
VideoMMMU
MMVU
VideoHolmes
Avg.
1
58.1
67.9
45.1
57.0
5
59.1
68.8
46.7
58.2
10
57.1
65.0
38.2
53.4
Appendix
Table 7: Sensitivity of top- k in DRD.
Method
AIME24
AIME25
HMMT25
Avg.
Δ
Base
16.7
26.7
10.0
17.8
–
Mixed-Policy
36.7
40.0
16.7
31.1
+13.3
GRPO
40.0
40.0
20.0
33.3
+15.5
Echo-GRPO
53.3
46.7
23.3
41.1
+23.3
Appendix
Table 8: Effect of Echo-GRPO on text reasoning benchmarks.
Method
Avg. Perf. ( ↑ )
Avg. Clip ( ↓ ) (%)
0–100
100–200
200–300
300–400
400–500
500+
Base
50.5
–
–
–
–
–
–
–
Mixed-Policy
51.7 (+1.2)
26.1
11.2
9.0
21.5
44.6
42.1
28.2
GRPO
56.4 (+5.9)
7.8
9.6
8.4
8.2
6.9
7.7
5.7
Echo-GRPO
58.2 (+7.7)
7.9
10.2
8.6
7.3
7.5
8.1
5.5
Appendix
Table 9: Performance and clipping statistics during scaled-up training. The average performance covers three major benchmarks: VideoMMMU, MMVU, and Video-Holmes. Average Clip presents the average percentage of samples clipped throughout the steps.
Strategy
Avg.
Δ vs. GRPO
Offline cost †
GRPO
56.0
–
–
Echo-GRPO w/o DRD
56.9
+0.9
9.4 ms/token
Echo-GRPO w/ DRD
58.2
+2.2
163.9 ms/token
Appendix
Table 10: Performance vs. offline construction cost. Avg. denotes the mean performance across VideoMMMU, MMVU, and Video-Holmes. Idiolectal paraphrasing is performed once during data construction; it introduces no additional training or inference-time overhead.