Visual Set Program Synthesizer
Organizations: Department of Computer Science, University of Oxford · Department of Computer Science and Engineering, Hong Kong University of Science and Technology · Institute of Logic and Computation, TU Wien
Abstract
A user pointing their phone at a supermarket shelf and asking "Which soda has the least sugar?" poses a difficult challenge for current visual Al assistants. Such queries require not only object recognition, but explicit set-based reasoning such as filtering, comparison, and aggregation. Standard endto-end MLLMs often fail at these tasks because they lack an explicit mechanism for compositional logic. We propose treating visual reasoning as Visual Program Synthesis, where the model first generates a symbolic program that is executed by a separate engine grounded in visual scenes. We also introduce Set-VQA, a new benchmark designed specifically for evaluating set-based visual reasoning. Experiments show that our approach significantly outperforms state-of-the-art baselines on complex reasoning tasks, producing more systematic and transparent behavior while substantially improving answer accuracy. These results demonstrate that program-driven reasoning provides a principled alternative to black-box visual-language inference.
Figures & tables
| Model | Type | Set-VQA (PA/AA) | GQA-Set (PA/AA) | PTR-Count (PA/AA) |
| End-to-End Baselines | ||||
| Qwen2.5-VL-72B | End-to-End | N/A / 38.6% | N/A / 35.1% | N/A / 25.4% |
| CogVLM-17B | End-to-End | N/A / 37.5% | N/A / 34.2% | N/A / 24.0% |
| InternVL-2.5-26B | End-to-End | N/A / 36.4% | N/A / 33.7% | N/A / 23.1% |
| Qwen2.5-VL-32B | End-to-End | N/A / 36.1% | N/A / 32.8% | N/A / 23.9% |
| LLaVA-1.6-72B | End-to-End | N/A / 34.8% | N/A / 31.5% | N/A / 22.4% |
| Model | Params | Inf. Setting | Single-Step AA | Multi-Hop AA | Extrema AA | Exclusion AA | Overall AA |
| Qwen2.5 | 72B | St. | 46.2 | 31.5 | 29.8 | 25.1 | 38.6 |
| Aug. | 51.4 | 36.1 | 31.2 | 28.6 | 42.1 | ||
| LLaVA1.6 | 72B | St. | 42.1 | 28.6 | 24.5 | 21.0 | 34.8 |
| Aug. | 46.8 | 32.5 | 26.8 | 23.4 | 37.9 | ||
| Qwen2.5 | 32B | St. | 43.8 | 29.4 | 27.1 | 22.8 | 36.1 |
| Aug. | 47.5 | 33.8 | 28.5 | 24.9 | 39.4 |
| Model | Overall (AA) | Task#1 (PA/AA) | Task#2 (PA/AA) | Task#3 (PA/AA) | Task#4 (PA/AA) | Task#5 (PA/AA) |
| CASTER (Qwen) | 88.90% | 90.5% / 89.5% | 91.2% / 90.1% | 89.5% / 88.1% | 87.2% / 86.5% | 85.1% / 83.4% |
| CASTER (LLaVA) | 87.50% | 89.1% / 88.3% | 90.3% / 89.2% | 88.0% / 86.8% | 85.9% / 84.9% | 83.5% / 81.9% |
| RLVR (Qwen) | 74.50% | 78.2% / 75.1% | 79.9% / 76.3% | 71.3% / 68.2% | 65.0% / 61.7% | 63.8% / 59.5% |
| SFT (Qwen) | 58.20% | 61.3% / 55.4% | 63.8% / 60.1% | 55.1% / 51.5% | 49.8% / 44.3% | 47.2% / 41.1% |
| Qwen2.5-VL-72B | 21.30% | N/A / 19.8% | N/A / 22.5% | N/A / 15.1% | N/A / 9.8% | N/A / 11.2% |
| CogVLM-17B | 20.10% | N/A / 18.5% | N/A / 21.1% | N/A / 13.5% | N/A / 8.5% | N/A / 10.5% |
| Model | PA (mean±std) | AA (mean±std) |
| CASTER (Full) | 95.1(±0.4)% | 93.8(±0.3)% |
| w/ Type-Only Match | 91.2(±0.7)% | 89.5(±0.6)% |
| w/ Binary Reward | 85.4(±1.0)% | 84.0(±1.1)% |
| w/ Normalized Reward | 94.8(±0.5)% | 93.5(±0.4)% |
| RLVR | 79.5(±1.2)% | 86.1(±0.9)% |
| Logic Operation | Depth | Std. PA | Std. AA | Logic Gap | Perc. Gap |
| FILTER (Color) | 1 | 98.1 | 96.8 | 1.9 | 1.3 |
| FILTER (Shape) | 97.9 | 96.5 | 2.1 | 1.4 | |
| FILTER (Texture) | 97.0 | 95.2 | 3.0 | 1.8 | |
| FILTER (Brand/OCR) | 95.5 | 92.4 | 4.5 | 3.1 | |
| COUNT (Basic) | 98.5 | 97.1 | 1.5 | 1.4 | |
| EXISTS (Basic) | 99.1 | 98.2 | 0.9 | 0.9 |