A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement

Yu Liu; Youxing Li; Yuedong Tan; Zhiwei Yang; Zihan Su; Hanqi Feng; Shaojie Zhang; Libin Liao

cs.AIOct 28, 2026

A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement

Authors: Yu Liu, Youxing Li, Yuedong Tan, Zhiwei Yang, Zihan Su, Hanqi Feng, Shaojie Zhang, Libin Liao,

Abstract

Recursive self-improvement requires models not only to acquire useful training experience, but also to decide when their current experience has ceased to be useful. Existing methods often focus on what experience to generate next, while paying less attention to when fresh experience should be introduced. Many also construct training targets through self-consistency or majority voting, making supervision depend on the model's own uncertain predictions. We address these two questions separately: training dynamics determine when to acquire new experience, while the answers to new tasks remain independently checkable. A-RSI monitors reward variation and accuracy in training logs to refresh a task pool when its learning signal declines, without consulting validation or test scores. A static library of model-authored programs produces multimodal questions and derives their reference answers; we specify the task-generation contract, automatically verify chosen properties of the reward and refresh rules, and separately audit the generated answers. The training loop uses no external training examples or human-written answer labels. On the primary eleven-benchmark suite, A-RSI raises the performance average from 57.77 to 61.05 with 800 fresh tasks.

Figures & tables

Figure 1: Motivation for

\mathcal{A}

-RSI . (a) Pool reuse can stall learning, while fewer truncations can raise accuracy. (b) Training logs guide refresh, and response audits distinguish completion from error correction.

Method	Task source	Supervision	Data updated	Selection rule
Generate tasks during training
Absolute Zero	Model-proposed code	Code execution	Continuously	Learnability reward
R-Zero	Challenger model	Model labels	Co-evolving	Near solver ability
MM-Zero	Multi-role model	Majority vote	Continuously	Difficulty reward
Vision-Zero	Image-pair games	Game rewards	Alternating	Self-play $\leftrightarrow$ RLVR
Select from existing tasks or distributions

Table 1: Comparison with related self-improvement and data-selection methods. Most change their training data continuously or at every step, whereas

\mathcal{A}

-RSI replaces the whole pool only when its own training logs show saturation and refills it with random programmatic tasks.

Figure 2: Overview of

\mathcal{A}

-RSI . Task templates, the candidate reservoir, and replacement pools are fixed before training. The controller reads training logs to decide when to activate a fresh pool; only the Solver policy is optimized.

Panel of Figure 3 — Figure 3: Training pools through $S_{32}$ , with columns aligned by continuation updates. Each numbered block represents 400 fresh tasks, and its width shows how long that pool is used. Pools are numbered in replacement order separately for each arm. Colors identify selection strategies, and triangles mark detector-triggered replacements.

Benchmark	$n$	$S_{0}$	Pool 1			Pool 2			Pool 3
			$S_{11}$	$S_{12}$	$S_{13}$	$S_{21}$	$S_{22}$	$S_{23}$	$S_{31}$	$S_{32}$	$S_{33}$
MMSI	1000	27.10	27.90	26.70	28.10	28.50	28.70	30.70	28.80	29.30	31.00
MathVerse	3940	62.03	62.01	62.39	62.46	63.40	64.06	63.38	63.25	63.48	63.81
MathVision	3040	45.92	47.27	47.20	47.66	48.98	50.07	50.62	51.45	51.45	52.53
MMMU-Pro (4)	1592	58.42	59.23	58.79	59.74	60.24	62.63	62.06	61.31	63.19	63.00
VisNumBench	1913	49.03	52.74	53.84	54.47	55.36	54.68	55.62	54.05	55.99	56.19

Table 2: Qwen3-VL-8B-Instruct on the eleven full benchmark sets (8192 tokens). Columns follow the

S_{ij}

notation defined in the text. Pool mean averages the three displayed columns within each pool before rounding.

Benchmark	$n$	$S_{0}$	Pool 1			Pool 2			Pool 3
			$S_{11}$	$S_{12}$	$S_{13}$	$S_{21}$	$S_{22}$	$S_{23}$	$S_{31}$	$S_{32}$	$S_{33}$
MMSI	1000	31.50	30.20	31.30	30.10	30.00	29.00	29.30	31.10	30.70	30.00
MathVerse	3940	70.10	72.34	72.23	72.11	71.88	71.80	71.42	71.83	72.39	71.93
MathVision	3040	53.95	54.51	55.69	55.46	55.62	55.82	56.35	56.78	56.41	55.30
MMMU-Pro (4)	1592	57.10	59.23	59.92	59.80	59.67	60.24	60.80	60.05	59.42	59.74
VisNumBench	1913	48.93	48.77	48.30	48.09	49.71	49.76	50.91	50.03	49.24	49.71

Table 3: MiMo-VL-7B-SFT on the eleven full benchmark sets (8192 tokens). Columns follow the

S_{ij}

notation defined in the text. Pool mean averages the three displayed columns within each pool before rounding.

Figure 4: Refresh policies and training feedback. (a) Qwen Macro11 on the eleven-benchmark suite (8192 tokens), with window 0 at the initialized checkpoint

S_{t}

. (b) Within-pool signal retention. Paired points compare models against the 85% signal threshold. Open markers denote final-round replay. The controller also checks accuracy gains.

Variant	Refresh rule	New-pool selection	$S_{21}$	$S_{23}$	$S_{32}$	New tasks by $S_{32}$
$\mathcal{A}$ -RSI	Training-log trigger	Random	59.84	60.40	61.05	800
No-Refresh	Never	None	58.48	58.35	–	0
Continuous	Every window	Adaptive difficulty	59.47	59.74	60.64	3,200
Periodic	Every two windows	Frontier	59.22	59.91	59.89	1,200
Frontier	Training-log trigger	Frontier	59.76	59.83	59.59	800

Table 4: Qwen refresh policies at matched updates. Scores are Macro11 on the eleven-benchmark suite (8192 tokens). New tasks exclude the shared initial pool. Differences are computed before rounding. A dash indicates an unavailable evaluation result.

Replacement selection	Macro11	Difference
Four windows, every-window refresh
Random control	60.29	–
Failure-prioritized sampling	57.67	$-2.63$
Learnable-band sampling	58.39	$-1.90$
Always-wrong sampling	57.89	$-2.40$
Ladder controller	59.47	$-0.82$

Table 5: Replacement-selection ablations. Scores are Macro11 at 8192 tokens on the eleven-benchmark suite. Each difference uses the control in its block and is computed before rounding.

Controlled check	Tasks passing
Correct committed answer ranks above incorrect committed answer	1,000 / 1,000
Canonical formatting raises reward with correctness and termination fixed	1,000 / 1,000
Normal commitment raises reward with correctness and formatting fixed	1,000 / 1,000

Table 6: Controlled reward scoring on 1,000 independently checked generated tasks. The second and third checks retain an incorrect answer, isolating auxiliary-reward increases without a correctness increase. Full response variants and component rewards are retained.

Figure 5: Training gains retained at twice the output budget. Gains over base on the eleven-benchmark suite are normalized to each method’s 4096-token gain (4.08 points for

\mathcal{A}

-RSI and 4.18 for MM-Zero v5). Base and trained checkpoints use the same output budget.

Panel of Figure 6 — Figure 6: From truncation to completion. Both base decodes reach the limit for each item. $S_{32}$ finishes correctly in both decodes: 1,759/3,437 tokens for A1 and 5,140/2,515 for A2. Shorter, completed responses can improve benchmark accuracy even when the reasoning remains flawed. These observations alone do not identify which reward component caused the change.

Panel of Figure 7 — Figure 7: From completed-but-wrong to completed-and-correct. Both independent base decodes give the same wrong answers (21.59 and 2), while both $S_{32}$ decodes give the correct answers (60.68 and 1). These corrections also persist in the seed-0 evaluations from $S_{21}$ through $S_{33}$ . Individual successes do not establish a net improvement over all originally completed items.

Strategy	Accuracy (gain)	Fresh tasks	Gain/1k	LJ	Trunc. %	Sentinel
$S_{t}$ (start)	57.77	0	–	54.83	14.34	WATCH
No-Refresh	Halted at $S_{31}$	0	–	–	–	–
Frontier	59.59 (+1.82)	800	+2.28	58.19	8.14	OK
Periodic	59.89 (+2.12)	1,200	+1.77	58.40	10.33	OK
Continuous	60.64 (+2.87)	3,200	+0.90	59.12	8.44	OK
$\mathcal{A}$ -RSI	61.05 (+3.27)	800	+4.09	59.54	4.16	OK

Table 7:

S_{32}

policy results on the eleven-benchmark suite at 8192 tokens. Parentheses give the gain over

S_{t}

; LJ averages judge accuracy over the same eleven benchmarks. Truncation is pooled over 22,042 items. The sentinel uses the 114 retained yes/no items.

Arm	Geometry (4,588)		Chart (2,960)		Table (1,181)		Counting (447)
Arm	Acc.	Trunc.	Acc.	Trunc.	Acc.	Trunc.	Acc.	Trunc.
$S_{t}$ (start)	56.19	18.64	79.63	1.69	70.96	13.80	54.81	13.20
Frontier	57.65	10.48	81.32	1.05	72.14	10.16	57.94	6.71
Periodic	57.85	12.69	82.23	1.11	73.58	11.35	59.06	10.07
Continuous	59.70	9.94	81.15	0.71	74.26	9.40	57.94	5.82
$\mathcal{A}$ -RSI	59.83	4.80	81.66	0.81	72.99	6.60	61.74	2.46

Table 8: Visual-type results at

S_{32}

, 8192 tokens. The fixed annotations are restricted to benchmarks retained in the eleven-set suite; the science MC sets have no visual-type annotations and are excluded from these type subsets. Accuracies and truncation rates are pooled within each type; bold marks the best refreshing arm.

Variant	Refresh rule	New-pool selection	$S_{21}$	$S_{23}$	$S_{32}$	New tasks by $S_{32}$
$\mathcal{A}$ -RSI	Training-log trigger	Random	59.84	60.40	61.05	800
No-Refresh	Never	None	58.48	58.35	–	0
Continuous	Every window	Adaptive difficulty	59.47	59.74	60.64	3,200
Periodic	Every two windows	Frontier	59.22	59.91	59.89	1,200
Frontier	Training-log trigger	Frontier	59.76	59.83	59.59	800

Table 9: Refresh-policy comparisons for Qwen3-VL-8B-Instruct on the full eleven-benchmark suite. Columns align each arm by the continuation updates needed to reach the indicated

\mathcal{A}

-RSI state. Values are Macro11 accuracy at an 8,192-token limit. New tasks count distinct introduced training instances, not total generation or compute. A dash denotes a missing accepted result; the no-refresh arm stopped at the training budget corresponding to

S_{31}

Experiment	How the replacement tasks are chosen	Accuracy	Stable
Probe selection	1–7 of 8 correct, image-specific failures first	$-2.63$	$-2.60$
	1–7 of 8 correct (ThinkLite-style)	$-1.90$	$-1.95$
	0 of 8 correct	$-2.40$	$-2.37$
Ladder control	pointer up at pass $\geq.80$ , down at $\leq.20$	$-0.82$	$-0.90$
	every pointer up one tier per window	$-0.48$	$-0.58$
	ladder controller with a truncation rule	$-0.05$	$-0.01$

Table 10: Replacement-selection differences on the eleven-benchmark suite, 8192 tokens, against the matched random arm. The first two blocks share the every-window random control after four windows; the last compares with

\mathcal{A}

-RSI after eight. Stable applies the same scoring to letter-, number-, and expression-gold items and equally averages the eleven subset accuracies. Differences are descriptive single-run estimates.

Model / training state	Accuracy			Truncation (%)
Model / training state	4096	8192	Gain	4096	8192
Base model	52.91	54.91	$+2.00$	20.10	14.42
MM-Zero v5 (released weights)	57.09	57.37	$+0.28$	2.84	0.87
$\mathcal{A}$ -RSI , $S_{32}$	56.99	58.16	$+1.17$	6.82	3.44

Table 11: Length-budget comparison on the eleven-benchmark suite: the public base model, released MM-Zero v5, and

\mathcal{A}

-RSI at

S_{32}

. Each row evaluates fixed weights at both limits; Gain is the 8192-token score minus the 4096-token score. Earlier continuation and curriculum results appear in Table 24 .

Qwen variant at matched $S_{23}$ training progress	Macro11
Original-pool continuation	58.35
Fresh random replacement ( $\mathcal{A}$ -RSI )	60.40
Replacement minus continuation	$+2.05$

Table 12: Fresh-pool training comparison on all eleven benchmarks, using accuracy scoring and an 8192-token limit at the same update budget. Macro11 and the difference are computed before rounding. Pool replacement also changes pool size from 500 to 400 tasks.

Generator group	Answer agreement	Audit scope
Multi-hop geometry (mv)	3,881 / 3,881	All candidates
Referring charts (ref)	1,568 / 1,568	All candidates
Tables, counting, single-series charts	4,580 / 4,580	All candidates
Plane geometry (ga)	150 / 150	Sample only
Multi-series chart reading (hr)	1,922 / 1,927	Audited subset; five tie ambiguities

Table 13: Independent reference-answer checks. Denominators are the audited records for each group, not all 16,000 reservoir candidates. Agreement is distinct from proof that the question is visually necessary or unambiguous.

Model	Pool	$U_{\mathrm{first}}$	$U_{\mathrm{last}}$	Retention	$\Delta a$	Action	Record
Qwen	Initial	239	195	81.59%	$+7.00$	Refresh	Executed
	Replacement 1	196	163	83.16%	$+9.47$	Refresh	Executed
	Replacement 2	181	152	83.98%	$+6.59$	Refresh	Replay
MiMo	Initial	245	153	62.45%	$+14.28$	Refresh	Executed
	Replacement 1	199	143	71.86%	$+14.50$	Refresh	Executed
	Replacement 2	152	150	98.68%	$+6.63$	Keep	Replay

Table 14: Training feedback and controller actions across all three pool stages. Retention is

U_{\mathrm{last}}/U_{\mathrm{first}}

, and

\Delta a

is in percentage points. Refresh requires retention at most 85% or an accuracy increase of at least ten points, after the minimum observation period.

Decoding	Total gain (Macro11)	Finished wrong $\to$ right	Finished right $\to$ wrong	Both-finished contribution
Primary	$+1.63$	1,460	1,235	$+0.99$
Repeat	$+2.21$	1,483	1,169	$+1.69$

Table 15: MiMo’s full eleven-benchmark output comparison,

S_{0}\to S_{32}

. Gains and contributions are percentage points; transition counts are totals across benchmarks. The contribution averages per-benchmark net transitions divided by each benchmark’s full test size. Both decodings use the same trained weights, so they are decoding repeats rather than independent training runs.

Model	Start	Decoding	Total gain	Both-finished contribution	Other strata contribution
Qwen	$S_{0}$	Primary	$+4.40$	$-0.75$	$+5.15$
Qwen	$S_{0}$	Repeat	$+4.85$	$-0.31$	$+5.16$
Qwen	$S_{t}$	Primary	$+3.27$	$-0.71$	$+3.98$
MiMo	$S_{0}$	Primary	$+1.63$	$+0.99$	$+0.64$
MiMo	$S_{0}$	Repeat	$+2.21$	$+1.69$	$+0.52$

Table 16: Full-suite Macro11 accounting through

S_{32}

, in percentage points. The Qwen

S_{0}

comparison includes initialization; the

S_{t}

comparison starts after initialization. Contributions add before rounding. Completion strata are post-training outcomes, so this decomposition does not identify a causal fraction of reasoning improvement.

Generator group	Tasks	Independent answer agreement
Multi-hop geometry (mv)	200	200 / 200
Referring charts (ref)	200	200 / 200
Single-series charts (ca)	200	200 / 200
Tables (tab)	200	200 / 200
Counting (sc)	200	200 / 200
Total	1,000	1,000 / 1,000

Table 17: Independent checks of the 1,000 generated tasks in

\mathcal{A}

. Membership is fixed before checking, without replacement of failed cases. Solvers use the recorded textual state, and image hashes check file identity.

Model	$\Delta_{\mathrm{sample}}$	$D_{11}$	Other strata	$\|\mathcal{I}_{11}\|$	Wrong $\to$ right	Right $\to$ wrong
Qwen	$+4.70$	$-1.20$	$+5.90$	834	52	64
MiMo	$+1.30$	$+1.30$	$0.00$	901	70	57

Table 18: Automated analysis of the fixed random 1,000-question sample,

S_{0}\to S_{32}

, at 8192 tokens. Score changes are percentage points over all 1,000 questions. The last two columns include only questions for which both responses finish; Other strata sums

D_{00}+D_{01}+D_{10}

. No result-conditioned filtering is applied.

Benchmark	$S_{t}$	$\mathcal{A}$ -RSI	Change
ChartQA	82.84	84.96	$+2.12$
MMMU	62.23	64.36	$+2.12$
MMSI	26.40	29.30	$+2.90$
MathVerse	62.26	63.48	$+1.22$
MathVision	47.04	51.45	$+4.41$
VisNumBench	52.90	55.99	$+3.08$

Table 19: Per-benchmark accuracy on the eleven-benchmark suite at 8192 tokens,

S_{t}

against

\mathcal{A}

-RSI at

S_{32}

. All eleven benchmark point estimates improve.

	v1	v2	v3 (this report)
Who writes tasks	a snapshot of the model; program computes answers	programs only	programs only
Pool	300 rows, 225 shared, 75 new slots	400 fresh tasks per window	400 tasks, replaced when the detector fires
Replacement	controller picks tiers	frontier tiers	random (fixed family shares)
When	every 2 windows	every window	log-based detector
Difficulty tiers	state space and controller	control and diagnosis	diagnosis and reporting only
Health gate	$\geq 99/100$	$\geq 99/100$	$\geq 97/100$

Table 20: Evolution of the design.

Escalation rule	Late-stage tiers	Held-out vs. uniform	Macro11 4096
no escalation	F2 / H1	$+1.27$	56.05
up only on saturation	H1, H2	$+0.06$	56.69
uniform	all ten tiers	$+0.00$	56.47
half by state, half uniform	H2	$-2.12$	56.02
writer fixed at $S_{t}$	up to H2	$-0.99$	55.90
re-select by state each time	up to H3	$-2.42$	55.88

Table 21: Six-window chains from

S_{t}

. Benchmark scores use the eleven-benchmark protocol at 4096 tokens; held-out synthetic-task differences retain their separate original evaluation. The saturation-based arm has the highest benchmark point estimate.

Comparison	Gain		Reading
	4096	16k	Reading
Chains $-$ $S_{t}$ (mean of 7)	$+1.18$	$+1.17$	real capability
Uniform $-$ state-selected	$+0.74$	$+0.08$	only faster termination
Tool-assisted $-$ plain	$+1.26$	$-0.13$	only faster termination
Promotion on saturation $-$ no escalation	$+0.83$	$+0.68$	mostly kept, below noise

Table 22: Re-generating the earlier chains’ 4096-truncated answers at a 16k limit, under the original twelve-benchmark evaluation protocol. These are historical trajectories, not the eight-window

\mathcal{A}

-RSI run.

Arm	Macro11 at window (4096 tokens)
Arm	7	8	9	10	11	12
Partial refresh (225 of 300)	55.93	56.86	56.42	56.65	56.48	–
Fixed pool (300 tasks)	57.26	57.59	58.38	57.76	58.74	58.61
Partial refresh $-$ fixed pool	$-1.33$	$-0.72$	$-1.96$	$-1.10$	$-2.26$	–

Table 23: Extension run under the eleven-benchmark protocol at 4096 tokens, from the window-6 checkpoint (56.47). Partial-pool refresh has no accepted window-12 result because its health gate stopped training (98 of 100 normal updates, twice). Fixed-pool continuation has the higher point estimate at every shared window; its scores fluctuate rather than rising monotonically.

Checkpoint	Accuracy			Truncation (%)
	4096	8192	Gain	4096	8192
Base model	52.91	54.91	$+2.00$	20.10	14.42
MM-Zero v5 (released weights)	57.09	57.37	$+0.28$	2.84	0.87
Fixed-pool continuation, window 10	56.23	58.35	$+2.12$	12.60	6.83
Fixed-pool continuation, window 11	57.24	59.16	$+1.92$	10.16	4.55
Fixed-pool continuation, window 12	56.86	58.83	$+1.97$	10.27	5.21

Table 24: Supplementary length-budget comparisons on the eleven-benchmark suite, including earlier continuation and curriculum experiments. Earlier window indices belong to their original trajectories and do not denote the main

S_{ij}

states. Each row evaluates fixed weights at both limits.

Arm	Macro11 at window			Window 8 minus
Arm	4	6	8	$S_{t}$	No-Refresh
No-Refresh	60.97	60.19	60.01	$-0.34$	$+0.00$
Periodic	60.87	60.44	60.53	$+0.18$	$+0.52$
Continuous	61.04	61.07	60.69	$+0.34$	$+0.68$
$\mathcal{A}$ -RSI	60.40	61.04	60.71	$+0.36$	$+0.70$

Table 25: MiMo-VL-7B-SFT on the eleven-benchmark suite, Macro11 at 8192 tokens, one run per arm. The shared old-pool checkpoint at window 3 scores 60.35. Differences are computed before rounding from the available aggregate precision.

✓ Supported	✗ Not supported
A training log alone, with no test set, can tell that a pool is learned out. It lags the benchmark by about one window, fires on the old pool and on two replacement pools, and the half-life of a 400-task pool is three windows.	That the detector finds the best refresh time.
After the detector fires, random fresh tasks reach 61.05 with 800 new tasks, against 60.64 for replacing every window with 3,200 (4.09 against 0.90 points per thousand tasks).	That the $+0.40$ point estimate establishes a detector-timing advantage.
Choosing the replacement by difficulty frontier or by the model’s rollouts is not better than random (three experiments: $-2.63$ , $-0.82$ , $-1.45$ ).	That difficulty control or state-based selection improves the benchmark.
The old pool is learned out and the plateau is real: No-Refresh scores 58.48 and 58.35 before running out of signal; at window 6 every refreshing arm is 1.39 to 2.05 above No-Refresh .	That the pre-registered primary endpoint ( Frontier $-$ No-Refresh ) holds: No-Refresh halted, so there is no reading.
On Macro11, every-window random refresh $-$ original-pool continuation is $+1.82$ (stable $+1.66$ , LJ $+1.87$ ).	Any “significant” improvement; every arm was trained once.
Difficulty is quantifiable as monotone tiers and learnable in domain (tier 2 of one ladder .58 to .93).	Multi-hop training transfers specifically (MathVision $-0.30$ ; against a matched control $-0.37$ ).

Table 26: Claims from the current eleven-benchmark policy comparisons and explicitly scoped historical auxiliary experiments. Each numerical comparison uses its stated protocol.

Arm	4096 tokens								8192 tokens
Arm	$S_{11}$	$S_{12}$	$S_{13}$	$S_{21}$	$S_{22}$	$S_{23}$	$S_{31}$	$S_{32}$	$S_{12}$	$S_{21}$	$S_{23}$	$S_{32}$
No-Refresh	55.06	54.93	55.82	55.72	56.79	55.74	–	–	58.25	58.48	58.35	–
Frontier	55.06	54.93	55.82	56.74	56.74	56.94	57.41	56.83	58.25	59.76	59.83	59.59
Periodic	55.06	54.93	55.47	56.00	57.11	56.12	56.76	57.11	58.25	59.22	59.91	59.89
Continuous	54.74	54.93	54.86	55.85	55.77	55.92	56.90	56.82	58.14	59.47	59.74	60.64
$\mathcal{A}$ -RSI	55.06	54.93	55.82	57.33	57.63	58.18	58.45	59.50	58.25	59.84	60.40	61.05

Table 27: Macro11 at matched training progress on the eleven-benchmark suite; best values are bold. Two 4096-token evaluations of the same

S_{t}

weights score 54.55 / 54.72; they share the science-set outputs.

S_{t}

scores 57.77 at 8192 tokens. Shared prefixes reuse the same checkpoints.

Difference	$S_{32}$	$S_{23}$	$S_{21}$
Frontier $-$ No-Refresh	–	$+1.48$ ( $+1.37$ )	$+1.28$ ( $+1.16$ )
Frontier $-$ Continuous	$-1.05$ ( $-0.89$ )	$+0.09$ ( $+0.16$ )	$+0.29$ ( $+0.40$ )
Frontier $-$ $\mathcal{A}$ -RSI	$-1.45$ ( $-1.40$ )	$-0.57$ ( $-0.56$ )	$-0.07$ ( $-0.03$ )
Frontier $-$ Periodic	$-0.30$ ( $-0.26$ )	$-0.08$ ( $-0.04$ )	$+0.54$ ( $+0.53$ )
$\mathcal{A}$ -RSI $-$ No-Refresh	–	$+2.05$ ( $+1.93$ )	$+1.36$ ( $+1.18$ )
Periodic $-$ No-Refresh	–	$+1.57$ ( $+1.41$ )	$+0.74$ ( $+0.63$ )

Table 28: Eleven-benchmark pairwise accuracy differences at 8192 tokens, with the corresponding stable-format differences in parentheses. All differences use unrounded scores. No-Refresh has no

S_{32}

result because its training halted at

S_{31}

. These are point estimates from one training run per arm.

Benchmark	Frontier		Periodic		Continuous		$\mathcal{A}$ -RSI
Benchmark	Acc.	Trunc.	Acc.	Trunc.	Acc.	Trunc.	Acc.	Trunc.
ChartQA	84.40	0.20	85.96	0.28	84.00	0.32	84.96	0.56
MMMU	63.24	7.93	64.25	10.17	64.80	8.83	64.36	4.58
MMSI	28.10	3.30	29.30	4.10	27.00	2.10	29.30	2.50
MathVerse	63.68	3.78	63.81	5.00	65.94	2.18	63.48	1.93
MathVision	48.29	18.06	48.55	21.61	50.30	18.75	51.45	8.36

Table 29: Per-benchmark endpoint results on the eleven-benchmark suite at

S_{32}

and 8192 tokens. The best accuracy in each row is bold. Truncation is a percentage within each benchmark; the final row uses equal-benchmark accuracy and pooled-item truncation.

Family	Generator	Task type	Candidates	Difficulty tiers (diagnosis only)
Geometry	mv	MathVision-style multi-hop geometry; each hop uses the previous result and a new fact in the figure	3,881	two ladders of 5 tiers (1–5 hops)
Geometry	ga	plane geometry	3,880	5 tiers
Chart	hr	multi-series chart reading	2,091	ten-cell reading ladder
	ca	single-series chart (bar, line)	1,568	none
	ref	referring multi-hop chart	1,568	3 tiers
Table	tab	table	2,163	none

Table 30: The seven program-based generators and the candidate library. Gold answers were independently recomputed: mv 3,881/3,881, ref 1,568/1,568, tab/sc/ca 4,580/4,580, ga 150/150 on a sample, and hr 1,922/1,927 (the five differences are ties for “third highest”, 0.03% of candidates).

Panel of Figure 8 — Figure 8: Additional completed-answer corrections. Category aggregation and component-level calculation. C1 includes both relevant chart categories; C2 divides the branch voltage across its series resistors. Both decoding repeats show the wrong-to-right transition for each item; examples are selected qualitatively.

Panel of Figure 9 — Figure 9: Additional completed-answer corrections. Visual grounding followed by arithmetic. C3 distinguishes the two target fields before solving their scores; C4 counts the shaded pieces before multiplying by their area. Both decoding repeats show the wrong-to-right transition for each item; examples are selected qualitatively.

Panel of Figure 10 — Figure 10: Additional completed-answer corrections. Interpreting the marked geometry. C5 uses a supplementary rather than equal angle; C6 distinguishes standing-wave loops from full wavelengths in its final explanation. Both decoding repeats show the wrong-to-right transition for each item; examples are selected qualitatively.

Panel of Figure 11 — Figure 11: Additional completed-answer corrections. Visual comparison and symbol localization. C7 compares full object lengths; C8 locates the pianissimo marking in the requested bar. Both decoding repeats show the wrong-to-right transition for each item; examples are selected qualitatively.

Panel of Figure 12 — Figure 12: Additional completed-answer corrections. System scope and answer acceptance. C9 accounts for evaporation in every effect; C10 reads the chart label instead of treating a copyright date as a reason to abstain. Both decoding repeats show the wrong-to-right transition for each item; examples are selected qualitatively.

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 174 papers

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution
Oct 15, 2026Wei Dai, Qianchu Liu, Guanghui Qin +4Recursive Self-Improvement Rubric-Based RL
Massachusetts Institute of Technology · Microsoft Research · Johns Hopkins University
RSI-Master: Structuring Experiments to Guide Autonomous Model Improvement
Sep 28, 2026Yaxin Du, Xiyuan Yang, Zhifan Zhou +10Language Model Post-Training Self-Improving Agents
Shanghai Jiao Tong University · Carnegie Mellon University · University of Waterloo
Recursive Self-Improvement in Unified Multimodal Models
Oct 2, 2026Huijuan Wang, Chufan Shi, Cheng Yang +3Unified Multimodal Models Chart Understanding
University of Southern California · University of California San Diego · Carnegie Mellon University

Oct 15, 2026J/K move · Enter open · S save

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

Wei Dai, Qianchu Liu, Guanghui Qin, Xing Han+3

Massachusetts Institute of Technology · Microsoft Research · Johns Hopkins University

Recursive Self-Improvement Rubric-Based RL Reward Hacking

Open paper AskPDF arXiv

Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically. Open-ended expert tasks, like clinical report generations, have no answer key, which makes simple question-answer generation not viable. Here the proposer must also write a rubric that lists the content an answer should include and the errors it should avoid. However, the unique challenge behind model-written rubric is the comprehensiveness of it. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the learning process. To solve this problem, we introduce RRI, Recursive Reward Improvement, which uses a reward hacking adversary to supervise the reward. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win. The exploits found also rewrite how future tasks and rubrics are generated. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self-improvement baseline by 10.9 points at 9B and 12.4 at 27B, and by 10.3 points without any external model. In particular, the gains are largest on difficult and writing tasks. We show that the 27B model trained on the HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench without seeing a task from either. We further analyze the exploitable points of model-generated question-rubric pairs to facilitate future research on open-ended self-play.

Abstract

Language Model Self-Improvement

Explore similar work

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

RSI-Master: Structuring Experiments to Guide Autonomous Model Improvement

Recursive Self-Improvement in Unified Multimodal Models