Third Eye: Beyond Immediate Gains in Recursive Self-Improvement
Organizations: Mahindra University
Abstract
Does a useful self-update remain useful after the model updates again? Third Eye separates immediate performance from the outcome of a further native update. We evaluate five 3-8B backbones on math and code: 64 fresh runs, three candidates per run, and three continuations per candidate. Selection and final scoring use disjoint items. In the balanced Qwen3-4B/Llama-3.2-3B comparison, immediate-selected utility is near zero, but continuation reduces it by 1.78 percentage points (simultaneous 95% interval [0.68, 2.88]). Candidate-averaged immediate utility is already negative: selection partly accounts for the apparently neutral first step. Continuation also lowers utility for SmolLM3-3B and Llama-3.1-8B; Qwen3-8B remains inconclusive. Target and out-of-distribution losses dominate retention changes. Matched recipe and recipient-donor controls identify neither a reliable mitigation nor a single mechanism. The results support evaluating persistent self-updates as adaptation transitions, with independent scoring and a no-update baseline, rather than interpreting immediate performance as their full value.
Figures & tables
| Role | Math | Code |
|---|---|---|
| Training | GSM8K: 1,024 | MBPP: 306 |
| Dev. target | GSM8K: 64 | MBPP: 64 |
| Dev. OOD | MATH: 64 | MBPP val.: 64 |
| Final target | GSM8K: 128 | MBPP test: 128 |
| Final OOD | MATH-500: 128 | HumanEval: 128 |
| Retention | MMLU: 256 in each suite | |
| Selected | Uniform | |||
|---|---|---|---|---|
| Model | ||||
| Qwen3-4B | ||||
| Llama-3.2-3B | ||||
| SmolLM3-3B | ||||
| Qwen3-8B | ||||
| Llama-3.1-8B | ||||
| Declared contrast | Mean | Interval |
|---|---|---|
| Qwen4B Llama3B, | ||
| Pooled , these models | ||
| SmolLM3-3B, | ||
| Qwen3-8B, | ||
| Llama-3.1-8B, |
Language Model Self-Improvement
Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.
See all 192 papers