Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2

Nguyen Minh Thang; Trinh Thi Phuong

cs.AINov 4, 2026

Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2

Authors: Nguyen Minh Thang, Trinh Thi Phuong

Organizations: VNU University of Science, 334 Nguyen Trai, Thanh Xuan, Hanoi, Vietnam

Abstract

We study which training and calibration data help three open models solve ARC-AGI-2 tasks when the solver runs offline on four L4 GPUs for at most 12 hours. We compare data offline on public tasks and report 189 runs on the 120 hidden semi-private tasks. For the NVARC Qwen3-4B grid model, we built a corpus of 12,000 executable rules in a typed language and 95,993 tasks; a rule is admitted only if its behaviour on a fixed probe set differs from that of every admitted rule, and a task only if its pairs refute a shortlist of nearest competing rules and defeat 12 shortcut solvers. Continued fine-tuning on 1,920 real task records with revised four-GPU scheduling raised the semi-private score from 28.96 (mean of the released model) to 33.06 (mean of five runs, best 34.86); no checkpoint trained with synthetic tasks, even all 95,993 of them, improved on the real-task model. For the Qwen3.8-27B reasoning model, we fine-tuned on 1,036 of its own reasoning traces with correct final answers and compared sets that add traces of other models and synthetic tasks. With an inference procedure built around the model (a thinking cap enforced in the serving engine, a Python program checked on the demonstrations after each answer, a long first attempt per input), the score rose from 26.81 to 44.03, the level the ARC Prize Foundation measured for the base model without a time limit (42.36); most of the gain came from the inference procedure. For the Qwen3.8-Flash-Next mixture-of-experts model (125B parameters, 6B active), GPU memory limited concurrency. Pruning from 512 to 384 experts per layer by router-weighted activation saliency grew the key-value cache from 0.55M to 1.39M tokens; a 296-expert list adapted with ARC routing statistics, keeping experts for program tokens, plus an FP8 indexer cache, reached 1.96M. With a tool that checks programs, the model scored 56.67, our best result.

Figures & tables

Panel of Figure 2 — Figure 2: The admission funnel of the v2 rules ( Algorithm 1 ). Task generation stopped once each family had 1.25 times its quota of valid tasks, so only 18,635 of the admitted rules were attempted. The release removed aliases, kept each family’s quota and was audited independently; the export added up to seven tasks per rule from new salts.

Family	Stages	Programs	Candidates	Admitted	Attempted	Valid tasks	Released
mask pipeline	select $\to$ mask $\to$ 1–2 post ops $\to$ render	88.4M	24,000	17,132	6,975	3,206 (46%)	2,400
two objects	two objects $\to$ align $\to$ boolean relation	104,040	14,400	6,512	3,257	2,484 (76%)	1,800
multiple objects	group selector $\to$ aggregate	43,050	12,800	3,684	2,259	2,110 (93%)	1,600
panel boolean	two panels $\to$ align $\to$ truth table	50,400	9,600	4,076	1,538	1,538 (100%)	1,200
template symmetry	repair one defect of a symmetry	12,960	9,600	2,311	1,709	1,500 (88%)	1,200
path, ray, flood	rays, shortest paths, floods	9,360	8,000	2,102	1,613	1,307 (81%)	1,000

Table 1: Admission of the 10,000 v2 rules by family. “Admitted” rules passed the behavioural identity gate; “attempted” rules entered task generation, which stopped at 1.25 times the quota; the yield is relative to the attempted rules.

Set	Tasks	Input cells	Same shape	Colours/input	Tokens
		(median)		(mean)	(median / max)
v1 rules	15,200	810	21.8%	6	5,994 / 11,076
mask pipeline	18,240	810	22.6%	6	6,175 / 11,076
two objects	13,673	810	0%	6	5,882 / 6,707
multiple objects	12,160	810	0%	6	5,790 / 6,684
panel boolean	9,120	308	0%	6	2,647 / 3,020

Table 2: Statistics of the synthetic tasks (95,993-task export, training split) and of ARC-AGI-2. Cells and the share of pairs whose output has the input’s shape are per pair; tokens are per task in the compact format of NVARC 4B (worst dihedral view).

Figure 3: The first two of the six pairs of the base task of each corpus rule in Table 3 , drawn at one cell size per panel. Each input carries the 10-cell palette key on one border, and every pair draws a new palette, so the same rule paints with different colours from pair to pair.

Family (rule)	Rule	Algorithms
v1 select–mask–render (u2k_0053)	Select the object with the smallest anti-diagonal span, shift it one cell up, and draw the object above its shifted copy with a one-cell separator.	components, selector metric, shift, panel rendering
mask pipeline (u12k_03977)	Select the object with the smallest anti-diagonal span, take the cells that one step of dilation adds around it, keep those in exactly one of this halo and the object’s bounding box, keep the cells with exactly one occupied neighbour, and crop.	dilation, symmetric difference, endpoint detection
two objects (u12k_11778)	Take the object with the largest vertical imbalance and the topmost object, align their centres, and keep the cells of the second that the first does not cover.	two selectors, centre alignment, set difference
multiple objects (u12k_05969)	Take the three topmost objects, align their upper-right corners, and keep the cells that all three cover.	top- $k$ selection, alignment, per-cell consensus
panel boolean (u12k_07934)	Read the two framed panels, flip both top to bottom, align their lower-right corners, and mark the cells occupied in both.	frame detection, reflection, 4-bit truth table
template symmetry (u12k_09489)	Inside a frame, find the one cell whose change from the object to the auxiliary colour makes a cross-shaped region symmetric under a top-to-bottom reflection, repair it, and output its symmetry orbit.	symmetry orbits, unique single-cell repair

Table 3: One rule of each family. Families above the line are those of the 12,000-rule corpus (v1 and the seven v2 families); the last two are the families of the 72-program shape bank. “Object” means a connected component of the object colour.

Generator	Source of rules	Checks on rules and tasks	Distinct rules	Identifiable
re-arc	one hand-written generator per ARC-AGI-1 training task	examples checked by a per-task verifier	by design	not reported
BARC	generators written by a language model from seed programs	execution of the generated code	not reported	not reported
NVARC	puzzles and code written by a language model	execution of the code	not reported	not reported
Ours	points of a typed grammar, deterministic search	signatures on 220 grid probes and 352 typed slots, raw and dihedral, Hamming distance $\geq 3$ (two v1 pairs at 2); up to 8 shortlisted near rules and one-node variants refuted on $\geq 2$ pairs; 12 shortcut solvers fail; exact oracle replay	signatures	finite set

Table 4: Synthetic task generators for ARC. “Distinct rules” means that rules are checked to behave differently; “identifiable” means that each task is checked to separate its rule from a set of competing rules. “Not reported” means that we found no such check in the description of the generator.

Figure 4: Grid sizes of the synthetic corpus (547,158 pairs of the 95,993-task export, training split) and of ARC-AGI-2 (4,308 training and 530 evaluation pairs). Left and middle: the longer side of input and output grids. Right: the ratio of output to input cells on a

\log_{2}

scale; 0 means that the output has as many cells as the input.

Figure 5: Pairs of the training split of the synthetic export (91,193 tasks, 547,158 pairs) against ARC-AGI-2 training (1,000 tasks) and public evaluation (120 tasks). (a) Shape of the output relative to the input, by family; numbers are percentages of pairs. (b) Tokens per task in the compact format of NVARC 4B (worst dihedral view) for the synthetic rows of (a), with whiskers at 1.5 times the interquartile range and outliers not drawn; diamonds are the ARC-AGI-2 medians of Table 2 . (c) Share of changed cells in pairs whose output has the input’s shape. (d) Single-colour 4-connected components per input outside its most frequent colour, palette key included, clipped at 59. (e) Distinct colours per input.

Statistic	Synthetic	ARC-AGI-2 training	ARC-AGI-2 eval.
	(91,193 tasks)	(1,000 tasks)	(120 tasks)
Pairs per task (mean)	6	4.31	4.42
Output has the input’s shape	10.2%	65.9%	69.8%
Output smaller (neither side longer)	83.5%	24.9%	23.4%
Output larger (neither side shorter)	0.0%	8.6%	3.4%
Other shape changes	6.3%	0.7%	3.4%

Table 5: Transformation statistics of the synthetic tasks (95,993-task export, training split) and of ARC-AGI-2. Shares are of pairs unless the row says tasks; a sub-grid is a contiguous block of the input. Components are 4-connected single-colour regions of all colours except the input’s most frequent one; the synthetic count includes the cells of the palette key.

System	Draws	Mean $\pm$ SD	Best	$\Delta$
Base model (three runs of the released pipeline, one in our runtime)	4	28.96 $\pm$ 0.66	29.86	–
frozen base sampled without test-time training or search, Qwen3.8-27B selects	1	10.42	10.42	$-18.54$
ckpt-34000 or a checkpoint of the same run, continued on 1,920 real tasks	22	31.72 $\pm$ 1.52	34.44	$+2.77$
+ 25% FORGE50 synthetic episodes (one model)	1	29.31	29.31	$+0.35$
FORGE50 mixture with 86% synthetic episodes (one model)	1	28.47	28.47	$-0.49$
Base + 1,920 real + 95,993 rule-corpus tasks (one model)	1	28.61	28.61	$-0.35$

Table 6: Semi-private scores of the NVARC 4B line (ARC-AGI-2, 120 semi-private tasks, two attempts). Draws are separate runs; the pooled standard deviation of repeated runs of identical configurations is 1.94 points (27 draws of 9 configurations). Checkpoint lists come from the source code or log of the version that produced each score.

\Delta

is the difference of means to the base model. Rows marked

\dagger

ran in the same runtime as the row of ckpt-34000 with runtime changes, against which their differences are

-5.8

-6.5

(shape-balanced) and

-5.0

(real and synthetic).

Model	Init	Real	Synthetic	Views	Rank	Steps	Hardware, time
ckpt-34000	base	1,920	–	16	256	34,000	1 $\times$ L4, 10.0 h
FORGE50 continuation	ckpt-34000	1,920	FORGE50, 25% of episodes	16	256	45,000	rented GPU
mixed6000	base	1,090	4,910 rule corpus	2	64	1,500	RTX 3090 Ti
shape37090	mixed6000	1,090	36,000 shape-balanced	2	64	9,273	RTX 3090 Ti, 47.8 h
original7500	base	1,920	3,600 shape + 1,980 rule	16	256	120,000	RTX 5090
all-train	base	1,120	95,993 rule corpus	1	256	12,140	RTX 5090

Table 7: Offline fine-tuning runs of NVARC 4B. Real tasks come from ARC-AGI-1 and ARC-AGI-2; synthetic tasks come from FORGE50, the shape-balanced generator and the 12,000-rule corpus of Section 3 . All runs use rank-stabilised LoRA. Steps are optimizer steps of one task (rank 256) or of four tasks (rank 64).

Run	$\leq$ 16K	16–24K	24–32K	32–48K	48–56K	56–64K	forced at cap
Base, $T=1.0$ , 344 attempts	2/4	12/14	21/27	40/66	24/41	10/24	14/168
Base, $T=0.6$ , 172 attempts	2/2	9/12	9/12	25/36	5/11	4/16	3/83
RFT v1, $T=0.6$ , 172 attempts	2/2	12/14	7/11	19/33	7/18	9/16	7/78

Table 8: Accuracy of single attempts by thinking length on the public evaluation set (172 inputs per run, xhigh , 64K cap, 2-request harness: thinking, then a separate answer request). Columns are thinking lengths of natural completions; the last column counts forced completions at the cap. Entries are correct/attempts.

Figure 6: Single attempts of Qwen3.8-27B on the public evaluation set at a 64K cap (the runs of Table 8 ). (a) Share correct among natural completions, by thinking length, and among forced completions at the cap (crosses). (b) Attempts in each bucket (light) and correct attempts (dark).

Set	Composition	Rows	Eval rows	Thinking tokens
Answer-only	synthetic + official grids, thinking off	37,201	125	0
RFT v1	base xhigh self-traces	1,036 (1,033)	136 (61)	17.08M / 12.2K / 38.0K
main2	RFT v1 + second-pass traces at 88K	1,109 (1,103)	152	19.66M / 13.0K / 41.2K
short16k	traces of $\leq$ 16K tokens	617	4	4.07M / 5.7K / 13.4K
v3b	769 own traces + 1,284 explanations (1,000 synthetic) in the thinking channel	2,053	0	25.1M
ft64k	1,039 own traces + 1,602 explanations (1,584 synthetic) as answer text	2,641	–	20.2M + 17.9M

Table 9: Fine-tuning sets for Qwen3.8-27B. Rows are training examples (rows actually trained in parentheses), eval rows are rows from public evaluation tasks (tasks in parentheses), and thinking tokens are total / median / 90th percentile in Qwen tokens. All sets except the answer-only one contain only traces whose final grid equals the label.

Set	Channel of the synthetic rows	Synthetic	Corpus	Bank	Other rows	Result
v3b	thinking (explanations)	1,000	205	795	284 official explanations, 769 self-traces	9 ∗
ft64k	answer text (explanations)	1,584	1,412	172	18 official explanations, 1,039 self-traces	43
FINAL	thinking (teacher traces)	28	10	18	1,796 traces of official tasks	44
All		2,231	1,435	796

Table 10: Synthetic tasks in the fine-tuning data of Qwen3.8-27B. “Corpus” tasks come from the 95,993-task export of the rule corpus, “bank” tasks from the 72-program shape bank. The result is inputs solved of 172 by one 32K attempt, against 51 for RFT v1 in the same harness; ∗ 2-request harness, against 37 for RFT v1.

Model	Offline setting	Inputs	Task	H	Semi-private setting	SP
Base	two 64K attempts, $T=1.0$	84	49.86	–	two 64K attempts	26.81
Base	88K and 64K attempts	89	53.33	–	two 88K attempts	18.89
Base	two 64K attempts, program after each	–	54.03	–	program in a second request	21.39
RFT v1	one 32K attempt, grid or program	51	30.69	10	two 32K attempts, programs	37.22
RFT v1	one 48K attempt, grid or program	56	33.06	11	48K and 32K attempts	37.08
RFT v1	one 64K attempt, grid or program	62	39.03	15	–	–

Table 11: Qwen3.8-27B models offline without a time limit (public evaluation set, 172 inputs, Colab) and on the semi-private set (4

\times

L4, 12 hours). Offline rows give inputs solved and the task score; “grid or program” counts an input as solved if the direct grid or a train-perfect program output is right. H is the development subset of 45 inputs. a Mean over four pairings of one 64K attempt with four separate 32K draws (74, 73, 71 and 68 inputs), each pair ranked by the production rule; the 32K draws differ in weights (NVFP4 or BF16), GPU and seed, and none uses the reordered demonstrations of the deployed second attempt. c Half of the tasks answered by design. d Trained on H and on all evaluation tasks, so its offline numbers are in-sample.

Model	Harness	Metric	32K	48K	64K
Base FP8, $T=0.6$	2-request	grid	–	–	10 / 57
Base FP8, $T=1.0$ (two attempts)	2-request	grid	–	–	13–17 / 59–64
RFT v1	2-request	grid	8 / 37	–	13 / 63
v3b	2-request	grid	2 / 9	–	–
RFT v1	1-request	grid	8 / 40	9 / 50	14 / 54
RFT v1	1-request	two-slot	10 / 51	11 / 56	15 / 62

Table 12: Single-attempt accuracy of Qwen3.8-27B models on the public evaluation set at fixed thinking caps (inputs solved, of 45 for the subset H and of 172 overall). The 2-request harness thinks and then answers in a separate request (FP8 weights for the base model, BF16 for fine-tuned models); the 1-request harness is the production request of Algorithm 5 (NVFP4 weights). “Grid” counts direct answers; “two-slot” also counts inputs whose train-perfect program output is right. FINAL was trained on H tasks.

Engine, layout and cache	Cache tokens	Running	Tokens/s
stock vLLM 0.30, TP4, BF16 cache	94,001	4	164 (whole run)
Merlin 2.5.2, TP4, BF16 cache	96,743	–	137 (steady)
Merlin 2.5.2, TP4, FP8 cache	170,393	–	136–168 (steady)
Merlin 2.5.2, PP4, FP8 cache	255,273	–	136 (steady)
Merlin 2.5.2, TP2 $\times$ PP2, FP8 cache	420,026	12.6	249–256 (steady)
+ four draft tokens, no prefix caching, 96K context	466,151	16.7	301 (first 150 windows)

Table 13: Serving the unpruned Qwen3.8-Flash-Next (512 experts per layer) on four L4 GPUs. TP and PP are the tensor- and pipeline-parallel degrees; all layouts use expert parallelism. Throughput is total generated tokens per second, measured as noted: over the whole run, as the steady per-minute rate after the first five minutes, or over the first 150 ten-second windows of a probe; rows with different definitions are not directly comparable.

Set and setting	Metric	Qwen3.8-Flash-Next	Qwen3.8-27B
Public evaluation, 172 inputs, two 64K attempts	task score	49.31	49.86
	inputs solved	80	84
	completion tokens per attempt	55,057	53,756
Dev panel, 48K, two seeds	attempts correct (of 164)	29	29
Dev panel, 96K, one seed	inputs solved (of 82)	30	13
	thinking tokens per attempt	68,541	71,592

Table 14: The unpruned Qwen3.8-Flash-Next against Qwen3.8-27B offline. The full-evaluation run of Qwen3.8-Flash-Next used INT8 attention and Gated DeltaNet weights, 12 concurrent requests and a BF16 cache; the base Qwen3.8-27B used FP8 weights, an FP8 cache and 32 concurrent requests. Dev-panel runs compare against RFT v1 in NVFP4.

List	Experts	Checkpoint	Weights/GPU	Pin	Cache tokens	Tokens/s	Dev
	per layer	(GiB)	(GiB)	(GiB)		(150 / full)
unpruned	512	115.22	16.9 / 17.0	2.3	547,542	342 / –	78
REAP-448	448	–	15.1 / 15.5	–	–	–	26 a
REAP-384	384	100.38	13.3 / 13.7	5.8	1,391,054	521 / 433	87
CORE-320	320	92.96	–	6.7	1,605,632	561 / 463	84
CORE-300	300	–	–	–	–	–	77

Table 15: Expert lists for Qwen3.8-Flash-Next. Checkpoint size and weights per GPU (pipeline stage 0 / stage 1) are from the serving logs; the cache pin is the memory reserved for the key-value cache, set to the largest value that booted. Throughput is total tokens per second on the 60-task L4 probe over the first 150 windows and over the full probe. “Dev” counts correct attempts of 246 on the dev panel at 96K (three seeds). a One seed, of 82. b Two seeds, of 164. c With an FP8 indexer cache, which by the cache formula of the serving logs holds about 6% more tokens at equal pin; the measured gain of 11% for s288p also includes a larger pin. c8

-

40 and c8

-

72 remove from c8 the experts with the lowest

F_{e}

that are neither code experts nor specialists.

Figure 7: Expert lists of Qwen3.8-Flash-Next. (a) Correct attempts of 246 on the dev panel at a 96K cap by experts per layer and by the criterion that built the list (REAP-448, measured on one seed only, is left out); the dashed line is the unpruned model. (b) Key-value cache tokens and throughput over the first 150 windows of the L4 probe ( Table 15 ); open circles are lists served with the FP8 indexer cache. (c) Jaccard overlap of the kept experts, averaged over the 48 layers. (d) What c8 does with each of the 512 experts of each layer: experts outside REAP-384, experts of REAP-384 that c8 drops, the REAP-288 core that c8 keeps, the 753 REAP-288 experts that s288p exchanged out and the experts it took in their place, and the 8 experts per layer that c8 adds.

Quantity	512 experts	Pruned
Cache tokens, stock vLLM / Merlin 3	94,001 / 547,542	1,957,095 (c8)
Running requests on the probe	4 / 18.9	56
Tokens per second, first 150 windows (Merlin 3)	342	521 (REAP-384), 566 (s288p)
Tokens per second, full 6,000 s probe	–	433 (REAP-384), 492 (s288p)
Thinking tokens per attempt, offline, 96K cap	69.3K	68.5K (REAP-384)
Thinking tokens per twelve-hour preview (120 tasks)	–	17.7M direct, 15.6M agent

Table 16: Tokens before and after expert pruning. Rows not marked as offline or simulated are measured on four L4 GPUs; “simulated” rows come from our run simulator for about 343 hidden test inputs. Offline thinking lengths are on the dev panel at a 96K cap, pooled over three seeds.

Experts	Configuration (change from the previous row)	SP
512	stock vLLM, effort low , 40K	8.61
512	same, attempts of 24K and 40K	6.11
512	Merlin 3, xhigh , 96K / 48K, client timeout of 7,200 s	25.00
384	REAP-384	32.64
288	s288p, timeout removed	44.17
288	64K / 48K, second attempts ordered by confidence, FP8 indexer cache, $T=1.0$	40.97

Table 17: All runs of Qwen3.8-Flash-Next on the semi-private set, in the order they were made. Caps are thinking caps for attempts 1 and 2. Runs from the third row on use Merlin 3, four draft tokens and

T=0.6

unless noted; “1-input” and “multi” are tasks with one or with several test inputs.

Training or calibration data	Metric	Result	Control
NVARC 4B grid model ( Table 6 ); control: the released model unless stated
1,920 real tasks, continued fine-tuning	SP	31.72 $\pm$ 1.52 (22)	28.96 $\pm$ 0.66 (4)
then 25% FORGE50 episodes	SP	29.31 (1)	parent: 31.72 $\pm$ 1.52 (22)
86% FORGE50 episodes, 12% real and 2% hand-made tasks	SP	28.47 (1)	28.96 $\pm$ 0.66 (4)
1,920 real and 95,993 corpus tasks	SP	28.61 (1)	28.96 $\pm$ 0.66 (4)
1,920 real and 5,580 synthetic tasks	SP	28.06 (1)	28.96 $\pm$ 0.66 (4)

Table 18: Data experiments across the three models. Each row compares a training set (NVARC 4B, Qwen3.8-27B) or the data that choose the experts to keep (Qwen3.8-Flash-Next) with a control. The two sides share the evaluation; unless a row says “same code” or “same configuration”, or uses the dev panel, they can also differ in training length, initialisation or runtime, so most rows compare configurations rather than isolate the data. SP: semi-private score, mean

\pm

standard deviation over draws, with the number of draws in parentheses. Offline: test inputs solved of the 172 public evaluation inputs by one attempt at the given cap, in the 1-request two-slot harness unless marked 2-request. Dev: correct attempts of 246 on the dev panel at a 96K cap (82 inputs

\times

three seeds). The answer-only control was measured by the ARC Prize Foundation under its own protocol.

Family	Runs	Mean	SD	Median	Best
NVARC 4B pipeline with the base checkpoint	7	26.59	3.46	28.47	29.86
the released pipeline, unchanged	3	29.12	0.70	29.03	29.86
NVARC 4B pipeline with our checkpoints	136	29.57	2.77	29.44	34.86
Qwen3.8-27B, without our fine-tuning	6	22.66	3.51	22.09	26.81
Qwen3.8-27B, fine-tuned by us	11	32.35	6.90	33.61	44.03
NVARC 4B and Qwen3.8-27B in one run	3	27.41	15.35	31.53	40.28

Table 19: All runs on the semi-private set, grouped by method. The family rows count runs that scored 10 or more; the last two rows count, across families, runs that returned no score or 0 (one of them a Qwen3.8-Flash-Next run that exceeded the time limit) and completed runs that scored below 10 (five NVARC 4B runs, three Qwen3.8-27B runs including the answer-only fine-tune at 2.08, and the two Qwen3.8-Flash-Next runs at effort low with stock vLLM, 8.61 and 6.11), so the family means are conditional on a score of at least 10. “Without our fine-tuning” includes one third-party fine-tune of the same base model.

Model	Configuration (change from the previous row)	SP
answer-only fine-tune	thinking off, 8 dihedral views, vote	2.08
base	thinking at low effort, end-of-turn masked	9.86
base	xhigh , 64K cap on both attempts	26.81
base	+ program in a second request, $T=0.6$ , reordered second attempts	21.39
base	88K cap on both attempts	18.89
RFT v1	32K cap on both attempts, $T=0.6$ , two requests per attempt	28.19

Table 20: Selected runs of the Qwen3.8-27B line and the factor that changed at each step. Runs from 26.81 onward use four L4 GPUs, tensor parallelism 4, MTP and an FP8 key-value cache. Rows below the rule are comparisons against the 44.03 run.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

List (experts)	Construction	Measured result	Why rejected
REAP-448 (448)	public REAP list	dev 26 of 82 (one seed) against 30 unpruned; CUDA-graph capture out of memory at a 4.0 GiB cache pin	frees less memory, not more accurate
REAP-288 (288)	public REAP list	dev 66 against 78 unpruned and 87 REAP-384	accuracy
ARC-288 (288)	top 288 by ARC routing frequency $F_{e}$	dev 58, lowest of all lists (23 gained, 43 lost against unpruned, $p=0.019$ ); 85% natural completions at 62.6K thinking tokens	accuracy
CORE-320 (320)	REAP-288 + 32 experts by $F_{e}$	dev 84; fresh panel $+2.64$ points against REAP-384, lower bound $-4.46$	failed the adoption rule
CORE-300 (300)	REAP-288 + 12 experts by $F_{e}$	dev 77	accuracy
c336o (336)	CORE-320 + the 16 highest- $F_{e}$ experts that REAP-384 drops	dev 56 against 60 for REAP-384 (two seeds, 164); $+4.3\%$ throughput	speed gain below 5%

Appendix

Table 21: Expert lists and smaller checkpoints of Qwen3.8-Flash-Next that we rejected. “Dev” is correct attempts on the dev panel at a 96K cap (246 with three seeds unless noted); “production mix” is the task-weighted score of thinking and programs as used on the L4 GPUs, and “thinking” the score of the thinking alone, both as differences in points against c8. The adopted lists were REAP-384, then s288p, then c8 ( Table 15 ).

ARC-AGI

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) benchmarks the ability to infer and apply abstract rules from sparse examples.

See all 26 papers

Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2
Jun 30, 2026Johan LandVisual Reasoning Multimodal Reasoning
Independent Researcher
Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1
Jul 7, 2026Kabir Moghe, Peter ChinLLM-Based Program Synthesis Abstract Reasoning
Department of Computer Science Dartmouth College
OpenThoughts-Agent: Data Recipes for Agentic Models
Jun 23, 2026Negin Raoof, Richard Zhuang, Marianna Nezhurina +47Training Data Curation LLM Agent Training

Jun 30, 2026J/K move · Enter open · S save

Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2

Johan Land

Independent Researcher

Visual Reasoning Multimodal Reasoning ARC-AGI Multimodal CoT Reasoning

Open paper AskPDF arXiv

Large language models can produce fluent, internally coherent reasoning traces for abstract reasoning tasks while still being confidently wrong - making selection among candidates, not just generation, the central challenge. I present a solver for ARC-AGI-2, a few-shot visual reasoning benchmark, built around two principles: (i) treating reasoning modalities as search operators, generating diverse candidates independently across text, image, and code channels, and (ii) context-preserving holistic judging, in which a judge model jointly compares all candidate reasoning traces within a single long-context prompt. Unlike self-consistency or majority voting, this approach reliably recovers correct minority hypotheses on tasks where the modal answer is wrong. On the ARC Prize semi-private evaluation set, the solver achieves 72.9 percent at USD 38.99 per task - the highest score on the verified leaderboard at the time of writing, exceeding the best standalone frontier models, GPT-5.2 Pro at 54.2 percent and Gemini 3 Pro at 54.0 percent, by +18.7 percentage points. On the public evaluation set, it achieves 76.1 percent at USD 19.69 per task. I release the full source code and document extensive negative results, including the finding that prescriptive prompting templates and iterative refinement systematically reduce hypothesis diversity and degrade performance.

Abstract

ARC-AGI

Explore similar work

Modality-Driven Search with Holistic Trace Judging for ARC-AGI-2

Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

OpenThoughts-Agent: Data Recipes for Agentic Models