Personal AI assistants hold the potential to evolve from digital interfaces into embodied companions capable of guiding users through complex physical activities. For these assistants to become integral to daily life, they must do more than identify objects; they must provide precise, step-by-step instructions that align with a user's real-time progress. While Multimodal Large Language Models (MLLMs) show promise in general visual understanding, their ability to deliver grounded, sequential guidance for fine-grained manipulation tasks remains largely unverified. In this paper, we choose the jigsaw puzzle as a strategic testbed for this capability. Unlike general object recognition, puzzle solving demands high-precision spatial reasoning, the ability to distinguish between minute geometric variations, and a rigorous adherence to sequential logic. We investigate this capability through PuzzleMate, a novel framework focused on jigsaw puzzle solving captured through an egocentric viewpoint. We deploy PuzzleMate in a user-in-the-loop study to evaluate how well state-of-the-art MLLMs perceive the current puzzle state and generate actionable next-step instructions. Our analysis reveals seven key bottlenecks that limit their effectiveness. Building on these insights, we propose a benchmark that enables systematic evaluation of MLLMs' reasoning capabilities for puzzle solving. Our findings reveal a substantial performance gap in current models like GPT-5.2 and Gemini-2.5-Pro; while these MLLMs are highly capable, they struggle to navigate the intricate reasoning and sequential logic essential for jigsaw puzzle assistance.
Figures & tables
Figure 1: Overview of the proposed egocentric assistive system and interaction. (a) High-level workflow of our egocentric assistant, PuzzleMate , where first-person video from smart glasses and user queries are streamed to an MLLM, hosted on a remote server, to generate grounded responses. (b) An example of a human–AI interaction during puzzle solving, illustrating step-by-step, context-aware assistance through natural language dialogue grounded in egocentric visual input.
Figure 2: Sample egocentric puzzle images used in our benchmark. Representative 6 × 8 puzzle instances featuring Iron Man , Hulk , and Thor are used in our evaluation, highlighting both the visual diversity and structural consistency of the benchmark puzzles.
Figure 3: Overview of representative benchmark tasks used to evaluate egocentric spatial reasoning in MLLMs. Shown here are two of the seven tasks in our benchmark: Task 1 (Piece–Slot Compatibility) evaluates whether a candidate piece fits a given gap; and Task 2 (Piece Orientation Verification) tests the model’s ability to determine whether the piece is correctly rotated for alignment. Together, these tasks probe complementary aspects of spatial reasoning under egocentric visual input.
Figure 4: Task 7 (Piece-to-Location Matching with Natural Language Reference). The model identifies the correct placement location for a given puzzle piece among multiple missing regions using natural language descriptions.
Table 1: Performance comparison of MLLMs across the seven benchmark tasks. ▲ denotes percentage improvement when using Aria relative to GoPro, while ▼ denotes performance degradation.
Figure 5: Per-piece prediction confidence visualization across the puzzle. The heatmap, overlaid on the puzzle image, visualizes the Gemini-2.5-Pro [ 24 ] model’s average classification accuracy for Task 1. Each cell corresponds to a puzzle region and reports the mean outcome across the two candidate pieces: 1 indicates both pieces are correctly classified, 0 indicates both are misclassified, and 0.5 indicates one correct and one incorrect prediction.
Figure 6: Qualitative comparison of model behavior across task formulations . The top row contrasts Task 4 and Task 5, and the bottom row contrasts Task 6 and Task 7. Constrained, identifier-based outputs in Task 4 and Task 6 often lead to incorrect predictions, whereas allowing concise natural language responses in Task 5 and Task 7 with a limited token budget (35 tokens) yields clearer and more accurate assistance. When the token limit is removed (ET), models generate verbose and ambiguous responses, reducing usability. Overall, the figure illustrates how output modality and token constraints critically influence MLLM performance in egocentric assistive settings.
Vision-Language(-Action) Models (VLMs) are increasingly applied to interactive environments, yet existing benchmarks often overlook the complex physical reasoning required for point-and-click puzzle games. This paper introduces Vision-Language Against The Incredible Machine (VLATIM), a benchmark designed to evaluate human-like logical problem-solving capabilities within the classic physics puzzle game The Incredible Machine 2 (TIM). Unlike existing benchmarks, VLATIM specifically targets the critical gap between high-level logical reasoning and continuous action spaces requiring precise mouse interactions. This benchmark is structured into five progressive parts, assessing capabilities that range from basic visual grounding and domain understanding to multi-step manipulation and full puzzle solving. Our results reveal a significant disparity between reasoning and execution. While large proprietary models demonstrate superior planning abilities, they struggle with precise visual grounding. Consequently, they do not yet show human-like problem-solving capabilities.
Maximilian Triebel, Marco Menner, Dominik Helfenstein
Institute of Artificial Intelligence, University of Stuttgart, Stuttgart, Germany.
Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that create ambiguous ground truth in texture-repeated regions. We introduce \ours{}, a benchmark with tab-and-blank interlocking pieces where geometric constraints provide strong local compatibility requirements that, combined with visual content, yield unambiguous ground truth. Across 95K instances at four grid densities (4×4 to 16×16), we find that zero-shot VLMs largely lack geometric reasoning: only one of five frontier models (GPT-5.5) exceeds random baseline on 4×4 puzzles, while all others perform at chance level. While supervised fine-tuning achieves >97% on 4×4, all models collapse on larger grids: GPT-5.5 drops from 70% to near-random on 8×8, and even fine-tuned models fall below 5% on 12×12. This ``scaling cliff'' suggests current architectures cannot maintain consistent constraint satisfaction as the number of pieces increases. \ours{} establishes scalable geometric reasoning as an open challenge for vision-language models.
Shawn Li, Wei Yang, Jike Zhong +11
University of Southern California · National University of Singapore · Adobe Research +3
We introduce Pencil Puzzle Bench, a framework for evaluating large language model reasoning through pencil puzzles, a family of constraint-satisfaction problems closely related to NP-complete problems, with deterministic, step-level verification. From a database of 62,231 puzzles across 94 varieties with verified unique solutions, we select a benchmark of 300 puzzles spanning 20 varieties and evaluate 51 models from 11 providers in two modes: direct ask (single-shot) and agentic (multi-turn with iterative verification). A key differentiator of our benchmark is that every intermediate board state can be checked against variety-specific constraints, localizing errors to the exact rule violated, providing the infrastructure for dense, per-move reward signals for process supervision and reinforcement learning. Our evaluation reveals two distinct axes of capability: (1) reasoning effort scaling, where GPT-5.2 improves 81x from no reasoning to maximum effort; and (2) agentic iteration, where Claude Opus 4.6 rises from 0.3% to 30.0% through iterative checking, while GPT-5.2@xhigh improves from 20.2% to 56.0%. Agentic attempts span a median of 29 turns over 17 minutes, with the longest exceeding 1,221 turns and 14.3 hours - a demanding test of long-context utilization, not just reasoning.