RoSeViT: Role-Separated Vision Transformers for ARC Visual Reasoning
Organizations: University of Missouri - Columbia
Abstract
ARC tasks can be formulated as visual transformation problems, where a model must infer a high-level transformation rule from a small set of demonstrations and apply it consistently across a grid. While dense Vision Transformers (ViTs) perform well in this setting, they entangle local visual state and global transformation logic within a shared token representation, which can hinder consistent reasoning. In this paper, we introduce RoSeViT, a role-separated Vision Transformer that explicitly decouples high-level instruction inference from visual workspace computation. RoSeViT uses controller tokens to aggregate transformation-level information and workspace tokens to represent the visual grid, with a structured attention pass in which nonlocal workspace interactions are routed through the controller. This design is motivated by our formal analysis showing that enforcing a shared instruction representation can reduce the effective complexity of visual transformation modeling. RoSeViT outperforms dense ViT baselines under the standard VARC training and evaluation protocol for ARC. In a controlled backbone replacement setting, it improves ARC-1 accuracy from 54.5 to 56.6, and with recurrent refinement reaches 58.8. When integrated into the full system, RoSeViT-Ensemble improves performance from 60.4 to 62.6 on ARC-1. These results demonstrate that explicitly separating instruction and workspace representations leads to more effective visual reasoning on ARC tasks. GitHub repository: https://github.com/lz7fd/RoSeViT.
Figures & tables
| Model | #param | ARC-1 | GFLOPs theor. / actual | Latency |
|---|---|---|---|---|
| VARC-ViT (dense) | 18M | 54.5 | 53.8 / 53.8 | 20.7 ms |
| VARC-ViT, width-matched (dense) | 28M | 53.6 | 79.4 / 79.4 | 22.2 ms |
| RoSeViT d-1 (role-separated) | 28M | 56.6 | 75.9 / 97.3 | 46.1 ms |
| Inference condition (offline, exact match %) | RoSeViT d-1 | Dense 28M |
|---|---|---|
| intact | 81.3 | 77.9 |
| all conditioning zeroed (task + context / task) | 0.0 | 0.0 |
| wrong-task controllers | 2.9 | – |
| task token only zeroed | 4.3 | 0.0 |
| context tokens only zeroed | 5.5 | – |
| After TTT (ARC-1 pass@2, 2 seeds) | intact ablated |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Rule | Arm (seed) | 5k | 10k | 15k | 20k | 25k | 30k | 35k | 40k |
|---|---|---|---|---|---|---|---|---|---|
| T0 shortest path | RoSeViT (42) | .00 | .00 | .02 | .15 | .29 | .40 | .47 | .51 |
| RoSeViT (43) | .00 | .00 | .00 | .06 | .24 | .31 | .35 | .34 | |
| Dense (42) | .00 | .00 | .00 | .00 | .00 | .00 | .02 | .02 | |
| Dense (43) | .00 | .00 | .00 | .00 | .00 | .00 | .00 | .00 | |
| T1 wavefront-S | RoSeViT (42) | .00 | .00 | .01 | .01 | .01 | .02 | .02 | .05 |
| RoSeViT (43) | .00 | .00 | .00 | .00 | .01 | .00 | .02 | .02 |
Visual Reasoning
Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.
See all 170 papers