AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning

Sida Li; Yuanxin Liu; Linli Yao; Shicheng Li; Xu Sun; Haoli Bai; Xiaohui Li

cs.AIOct 9, 2026

AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning

Authors: Sida Li, Yuanxin Liu, Linli Yao, Shicheng Li, Xu Sun, Haoli Bai, Xiaohui Li

Organizations: National Key Laboratory for Multimedia Information Processing, Peking University · Huawei Technologies Co., Ltd.

Abstract

While reinforcement learning with verifiable reward (RLVR) enhances reasoning in large language models, its efficacy in long-horizon multimodal tasks remains underexplored. Visual Sudoku puzzles pose a representative challenge, requiring multi-step deductive reasoning over both local constraints and global dependency chains. To this end, we introduce FG-Sudoku, a fine-grained benchmark categorized by grid size and required deductive steps. Evaluations reveal that frontier MLLMs fail on complex long-range dependencies (e.g., 9*9 puzzles). During post-training, this near-zero success rate causes severe reward sparsity, halting standard RLVR. We propose Adaptive Curriculum Group Relative Policy Optimization (AdaC-GRPO) to overcome this. Unlike rigid or prior-based schedules, AdaC-GRPO dynamically constructs training batches by tracking the model's "Zone of Proximal Development" via online rollout accuracy. Experiments demonstrate that AdaC-GRPO enables Qwen2.5-VL-7B to approach the performance of Gemini-2.5-Flash while bridging 62.2% of the overall accuracy gap to Gemini-2.5-Pro, with its generalizability further validated across InternVL-3.5 and Qwen3-VL.

Figures & tables

Figure 1: Overview of AdaC-GRPO. The policy dynamically selects FG-Sudoku groups within the Zone of Proximal Development (ZPD) based on online rollout accuracy, excluding trivially solved and reward-starved tasks.

Figure 2: Overview of the dataset, showing a representative Sudoku sample, model-measured difficulty steps across subsets, and the metadata structure of individual entries. Panel (B) reports Gemini-2.5-Flash’s pass@1 accuracy on each subgroup as a post-hoc empirical validation of our solver-derived difficulty hierarchy; correctness is determined by exact-grid matching. Higher bars generally correspond to easier subgroups.

Figure 3: Visualization of the solution trace for a 4×4 Sudoku puzzle. Each quadrant represents a distinct class of operations. Operations applied to the same cell are connected by straight-line links, and the colors change continuously from blue to red to indicate the temporal sequence of the reasoning steps.

Table 1: Benchmarking leading proprietary and open-weight multimodal models (Pass@1). Panel (a) reports per-tier results, while panels (b) and (c) report group-level averages. A, B, and C denote the 4

\times

4, 6

\times

6, and 9

\times

9 subsets, respectively; 1–4 denote difficulty tiers. Block C compares training methods under the same Qwen2.5-VL-7B backbone, SFT warm start, reward, and rollout budget. Bold denotes the best result within Block C, while underlined values indicate the best result across all models.

Figure 4: Trajectory of policy competence during curriculum-guided unit testing. The horizontal axis represents sequentially activated training subsets, while the vertical axis shows the rollout accuracy (

f=1/4

). The orange shaded area denotes the globally defined Zone of Proximal Development (ZPD) bounds

[\tau_{\text{low}},\tau_{\text{high}}]

Curriculum Granularity	A	B	C	Total
Coarse-grained (size only)	0.6484	0.5547	0.0117	0.4049
Fine-grained (ours)	0.8906	0.6485	0.2189	0.5860

Table 2: Ablation results. (a) Fine-grained tiers bridge the gap between grid sizes. (b) Partial reward fails on Group C. (c) Each trained condition uses one input form throughout training; cross-modal rows change only the evaluation input.

Method	Easy	Medium	Hard	Overall
ZebraLogicBench-ID ( Lin et al., 2025 )
Base	0.453	0.047	0.014	0.165
GRPO	0.656	0.063	0.014	0.235
AdaC-GRPO	0.625	0.109	0.042	0.250
CMM-Math ( Liu et al., 2024c )
Base	0.526	0.480	0.410	0.469

Table 3: Pass@1 after training GRPO or AdaC-GRPO from the same task-specific initialization. ZebraLogicBench uses structural-complexity tiers; CMM-Math uses its provided level labels.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Train Set	Level 1	Level 2	Level 3	Level 4	Total
A (4 $\times$ 4)	376	271	239	138	1,024
B (6 $\times$ 6)	324	245	234	221	1,024
C (9 $\times$ 9)	308	246	214	256	1,024
Test Set	Level 1	Level 2	Level 3	Level 4	Total
A (4 $\times$ 4)	85	97	53	21	256
B (6 $\times$ 6)	82	64	62	48	256

Appendix

Table 4: Exact train/test counts per difficulty tier in the FG-Sudoku dataset.

Hyperparameter	Value
Base Model	Qwen2.5-VL-7B
GPUs	4 $\times$ NVIDIA A800
Precision	bfloat16
Optimization Method	GRPO
Optimizer	AdamW
DeepSpeed Stage	ZeRO-3

Appendix

Table 5: Detailed training configurations and hyperparameters.

Strategy	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4
Random	0.894	0.794	0.566	0.476	0.476	0.219	0.210	0.083	0.130	0.034	0.000	0.000
Median	0.929	0.887	0.679	0.524	0.695	0.469	0.290	0.188	0.203	0.068	0.000	0.000
Argmax (Ours)	0.953	0.897	0.717	0.571	0.634	0.500	0.226	0.125	0.304	0.153	0.015	0.000

Appendix

Table 6: Selection-strategy ablation within the ZPD window (1-epoch training). Argmax delivers the strongest C-group performance, confirming that consolidation-first progression is preferable to undirected sampling for long-horizon curriculum learning.

Figure 5: Heatmap visualization of the selection-strategy ablation (same data as Table 6 ). Color encodes accuracy on a $0$ – $1$ scale; framed cells mark the column-best entry per tier.

$[L,U]$	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4
$[0.1,0.7]$	0.870	0.845	0.623	0.524	0.634	0.406	0.274	0.188	0.203	0.051	0.000	0.000
$[0.2,0.7]$	0.918	0.856	0.679	0.619	0.646	0.484	0.226	0.125	0.290	0.153	0.000	0.000
$[0.3,0.7]$	0.929	0.887	0.679	0.667	0.634	0.484	0.258	0.167	0.333	0.203	0.015	0.000
$[0.1,0.8]$	0.906	0.866	0.642	0.524	0.646	0.406	0.258	0.188	0.203	0.051	0.000	0.000
$[0.2,0.8]$	0.953	0.897	0.717	0.571	0.634	0.500	0.226	0.125	0.304	0.153	0.015	0.000
$[0.3,0.8]$ (Ours)	0.941	0.907	0.679	0.667	0.659	0.484	0.290	0.209	0.333	0.203	0.015	0.000

Appendix

Table 7: Sensitivity to the ZPD bounds

[\tau_{\text{low}},\tau_{\text{high}}]

(1 epoch per setting). The two highest-

L

configurations (

L=0.3

) achieve identical C-group average accuracy and outperform lower-

L

configurations across most tiers, indicating that the method is robust to the choice of

U

but benefits from a moderately aggressive lower bound.

Figure 6: Heatmap visualization of the threshold-sensitivity sweep (same data as Table 7 ). Color encodes accuracy; framed cells mark the column-best entry per tier. Within each tier, color variation across configurations is small, which is the visual evidence for robustness to the choice of $[\tau_{\text{low}},\tau_{\text{high}}]$ .

Seed	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4	Total
Seed-1 (main)	0.976	0.928	0.774	0.667	0.878	0.641	0.548	0.396	0.478	0.288	0.074	0.017	0.5859
Seed-2	0.965	0.918	0.755	0.667	0.890	0.656	0.532	0.396	0.435	0.305	0.074	0.000	0.5794
Seed-3	0.988	0.938	0.755	0.619	0.866	0.625	0.565	0.375	0.464	0.271	0.059	0.033	0.5807
Mean	0.976	0.928	0.761	0.651	0.878	0.641	0.548	0.389	0.459	0.288	0.069	0.017	0.5820
Std	0.012	0.010	0.011	0.028	0.012	0.016	0.017	0.012	0.022	0.017	0.009	0.017	0.0034

Appendix

Table 8: Per-tier and total Pass@1 accuracy of Qwen2.5-VL-7B + AdaC-GRPO across three independent training seeds. Bold marks the best (or tied best) score per column. Seed-1 (the configuration reported as main in Table 1a ) yields the highest total accuracy across the three runs.

Seed	A1	A2	A3	A4	B1	B2	B3	B4	C1	C2	C3	C4	Total
Seed-1 (main)	0.906	0.835	0.509	0.476	0.732	0.422	0.355	0.292	0.087	0.068	0.000	0.000	0.4271
Seed-2	0.871	0.856	0.472	0.429	0.659	0.391	0.387	0.250	0.072	0.051	0.000	0.000	0.4089
Seed-3	0.929	0.814	0.491	0.476	0.720	0.406	0.339	0.271	0.058	0.068	0.015	0.000	0.4193
Mean	0.902	0.835	0.491	0.460	0.704	0.406	0.360	0.271	0.072	0.062	0.005	0.000	0.4184
Std	0.029	0.021	0.019	0.027	0.039	0.016	0.024	0.021	0.015	0.010	0.009	0.000	0.0091

Appendix

Table 9: Per-tier and total Pass@1 accuracy of Qwen2.5-VL-7B + GRPO across three independent training seeds. Bold marks the best (or tied best) score per column.

Benchmark	Baseline	AdaC-GRPO
VsiBench ( Yang et al., 2025 )	0.3305	0.3527
MMVU ( Zhao et al., 2025 )	0.6352	0.6390
TempCompass ( Liu et al., 2024d )	0.6947	0.6839
ZebraLogic ( Lin et al., 2025 )	0.0340	0.0620
CrosswordBench ( Leng et al., 2025 )	0.051	0.075
Average reasoning length	1910.88	975.58

Appendix

Table 10: Transfer evaluation of the Sudoku-trained model. Values are mean accuracy, except for CrosswordBench (ICR) and reasoning length (tokens).

Figure 7: Comparison of the time cost between unit testing and subset training in each course-learning iteration. The unit-testing module introduces only a small overhead across different sampling ratios.

Figure 8: Accuracy of the unit-testing module under different sampling ratios, using ratio = 1 as the reference. Even with substantially reduced sampling (e.g., ratio = 1/8), the accuracy remains close to the full-data result.

Method	Discard	All-wrong	All-correct	Pass
DAPO	99.7%	99.7%	0.0%	0.0%
AdaC-GRPO	54.7%	47.3%	7.4%	31.5%

Appendix

Table 11: Rollout-group statistics on Group C with

G=8

. "Discard" denotes zero-variance groups; "Pass" is the trajectory-level pass rate.

Figure 9: Difficulty-Induced Entropy Bifurcation. The per-case mean entropy delta ( $\bar{H}_{post}-\bar{H}_{pre}$ ) reveals a distinct regime shift: predictive sharpening in easier tasks (negative $\Delta H$ ) versus uncertainty expansion in harder tasks (positive $\Delta H$ ).

Figure 10: Temporal Divergence from SFT Baseline. Jensen-Shannon divergence heatmap showing the magnitude of policy updates. High-complexity cases (bottom rows) exhibit sustained divergence, indicating significant exploration and trajectory re-parameterization by GRPO.

Figure 11: Stratified Entropy Distributions by Problem Scale. Comparisons of SFT and post-GRPO entropy across sizes 4, 6, and 9. The heavier-tailed distribution at larger sizes is consistent with increased uncertainty calibration on more complex reasoning paths.

Panel of Figure 12 — Figure 12: Selected excerpts from Gemini-2.5-Pro’s reasoning trace. Ellipses omit routine intermediate deductions, while highlighted regions mark state inconsistencies, explicit backtracking, and the final false-positive validation.

Explore similar work

DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage
Mar 1, 2026Haowen Gao, Zhenyu Zhang, Liang Pang +7Group Relative Policy Optimization RL for VLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Kuaishou Technology, Beijing, China
MaxSAT-Based Feedback for Guiding Vision-Language Models in Sudoku
Jul 14, 2026Pedro Orvalho, Guillem Alenyà, Felip ManyàVision-Language Models VLM Reasoning
Artificial Intelligence Research Institute (IIIA), Consejo Superior de Investigaciones Científicas (CSIC), Barcelona, Catalonia, Spain · Institut de Robòtica i Informàtica Industrial (IRI-CSIC-UPC), Barcelona, Spain
RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning
Feb 25, 2026Yukun Chen, Jiaming Li, Longze Chen +10VLM Reasoning Rubric-Based RL
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Alibaba Group +1

Mar 1, 2026J/K move · Enter open · S save

DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage

Haowen Gao, Zhenyu Zhang, Liang Pang, Fangda Guo+6

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Kuaishou Technology, Beijing, China

Group Relative Policy Optimization RL for VLMs Multimodal Reasoning Adaptive Sampling

Open paper AskPDF arXiv

Reinforcement learning (RL) with group relative policy optimization (GRPO) has become a widely adopted approach for enhancing the reasoning capabilities of multimodal large language models (MLLMs). While GRPO enables long-chain reasoning without a critic, it often suffers from sparse rewards on difficult problems and advantage vanishing when group-level rewards are too consistent for overly easy or hard problems. Existing solutions (sample expansion, selective utilization, and indirect reward design) often fail to maintain enough variance in within-group reward distributions to yield clear optimization signals. To address this, we propose DIVA-GRPO, a difficulty-adaptive variant advantage method that adjusts variant difficulty distributions from a global perspective. DIVA-GRPO dynamically assesses problem difficulty, samples variants with appropriate difficulty levels, and calculates advantages across local and global groups using difficulty-weighted and normalized scaling. This alleviates reward sparsity and advantage vanishing while improving training stability. Extensive experiments on six reasoning benchmarks demonstrate that DIVA-GRPO outperforms existing approaches in training efficiency and reasoning performance. Code: https://github.com/Siaaaaaa1/DIVA-GRPO

Abstract

Explore similar work

DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage

MaxSAT-Based Feedback for Guiding Vision-Language Models in Sudoku

RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning