Organizations: Independent Researcher · State Key Lab of CAD&CG, Zhejiang University · National Key Laboratory for Novel Software Technology, Nanjing University · SnT, University of Luxembourg · CSAIL, MIT / ESAT, KU Leuven
Building AI systems that can plan, act, and create in the physical world requires more than pattern recognition. Such systems must reason about the generative mechanisms and constraints governing physical processes, using structured representations that connect observations, actions, and their effects. Yet, many existing benchmarks study these capabilities separately, focusing either on visual recognition or on abstract symbolic or programmatic reasoning. Origami provides a natural testbed that integrates these abilities: constructing shapes through folds requires visual perception, reasoning about geometric and physical constraints, and sequential planning, while remaining sufficiently structured for systematic evaluation. We introduce OrigamiBench, a benchmark for evaluating programmatic understanding of the mechanisms underlying origami synthesis through a high-level language of physically grounded fold actions. Experiments with modern vision-language models reveal that scaling model size alone does not reliably improve reasoning about physical transformations. Moreover, models struggle to ground programmatic information in visual observations, suggesting that visual and language representations remain weakly integrated.
Figures & tables
Figure 1: Visual summary of OrigamiBench . On the left ( Data ), representative realistic, synthetic, and semi-synthetic examples are shown as program expressions paired with their rendered origami states. On the top right ( Environment ), a program statement is executed from the current state through the generative folding process: the compiler converts the statement into a crease-pattern representation, the folding validator checks the resulting state, and the renderer produces the rendered next state. At the bottom ( Tasks ), the two multiple-choice evaluations are illustrated from the model’s perspective. Task 1 ( Transition Reasoning ) provides the current state and candidate successors, whereas Task 2 ( Program Grounding ) additionally provides the generating program statement. Green and red marks indicate the correct and incorrect states respectively.
Fold
Name
Invocation
Expansion
Pleat
(pleat (X Y U W L))
(combine (fold (line X Y) V (L)) combine (fold (line U W) M (L)))
Crimp
(crimp (X Y U W L))
(combine (fold (line X Y) M (L)) combine (fold (line U W) V (L)))
Inside Reverse
(inrev (X Y U L N))
(combine (fold (line X Y) V (L)) combine (fold (line X Y) V (N)) combine (fold (line X U) M (L)))
Outside Reverse
(outrev (X Y U L N))
(combine (fold (line X Y) M (L)) combine (fold (line X Y) M (N)) combine (fold (line X U) V (L)))
Rabbit Ear
(rabear (X Y U W Z L))
(combine (fold (line X Y) V (L)) combine (fold (line X U) V (L)) combine (fold (line X W) V (L)) combine (fold (line X Z) M (L)))
Table 1: Library primitives and their expansion into base folding actions.
Figure 3: Execution of an origami action program that constructs a whale. Each column corresponds to one program statement. The statement is compiled into a crease graph (middle row), checked for foldability, and executed to obtain the updated rendered paper state (bottom row). Thus, statements in the language correspond directly to folding actions rather than individual crease edges. In particular, a single atomic action may induce several crease edges: statement 6, for example, produces two valley and one mountain crease.
Figure 4: Validity under increasing compositional complexity. Valid sample rates of action expressions and crease patterns across 1 to 8 folds, evaluated on 50,000 paired random samples (2,000 per seed across 25 random seeds). Shaded regions show mean ± standard deviation across 25 random seeds.
Subset
Trajectories
Statements per traj.
Manually Annotated
25
4 / 7.7 / 13
Randomly Generated
80
1 / 3.3 / 8
LLM-generated
80
2 / 4.8 / 8
Total
185
1 / 4.6 / 13
Table 2: Composition and program lengths of OrigamiBench . Program lengths are reported as minimum / mean / maximum number of statements across trajectories.
Figure 5: Construction of evaluation instances. Each row presents a multiple-choice next-state prediction instance, consisting of a reference origami state and four candidate successors. For programmatic grounding, the intervention is highlighted in white.
Figure 6: Accuracy on Task 1 (Transition Reasoning) across models for the Associative (Task 1a) and Causal (Task 1b) settings. The dashed line marks the 25% random-guess baseline.
Figure 7: Accuracy on Task 2 (Program Grounding) across the No Grounding (Task 2a), Weak Grounding (Task 2b), and Programmatic Grounding (Task 2c) settings. The dashed line marks the corresponding random-guess baselines. Detailed results for individual intervention types in Task 2c are reported in Appendix G.3 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Name
Semantic
Complexity
010_chan_Beaver
animals-aquatic
Easy
065_komatsu_Dolphin
animals-aquatic
Easy
030_tanakaoripa_fish_base
animals-aquatic
Medium
129_komatsu_Shore_Crab
animals-aquatic
Medium
221_winston_Cardinal_Tetra
animals-aquatic
Medium
226_kamiya_Hermit_Crab
animals-aquatic
Medium
Appendix
Table 3: Dataset Structure
Semantic Category
Easy
Medium
Hard
TOTAL
animal-aquatic
2
6
11
19
animal-bird
7
5
2
14
animal-insects
2
4
34
40
animal-mammals
18
21
4
43
animal-reptiles
0
1
5
6
animal-others
6
8
18
32
Appendix
Table 4: OrigamiBench Dataset Statistics by Semantic Category and Complexity Level
Table 6: Accuracy performance of different VLMs on the one-step multiple-choice evaluation task(8 options).
Figure 9: Examples for the one-step task. The top and the bottom three rows show Associative and Causal task examples, respectively. Additionally, three task examples with varying degree of complexity are shown for both Associative and Causal. Each task requires to choose the next folding step of the reference origami (first column) from four candidate options (A–D). The correct answer is highlighted using the green background. Coloured circles at the bottom-right corner of each option indicates the prediction made by each model.
Variant
Distractor source
Distractors
Options
Retained
Skipped
Task 2a
Other origami sequences
3
4
173
0
Task 2b
Same origami sequence
3
4
173
0
Task 2c–Mix
Mixed interventions
3
4
172
1
Task 2c–Point
Point intervention
3
4
172
1
Task 2c–Type
Mountain/Valley intervention
1
2
165
8
Task 2c–Layer
Layer intervention
1
2
9
164
Appendix
Table 7: Construction and statistics of the Task 2 variants. “Skipped” is measured relative to the 173 eligible source transitions. Task 2a and Task 2b use every eligible transition, whereas Task 2c retains a transition only when the required number of valid and visually distinct counterfactual distractors can be generated.
Figure 10: Accuracy across individual intervention types in Task 2c (Programmatic Grounding). Point, Type, and Layer modify the fold-line points, mountain/valley type, and layer selection, respectively. The dashed lines mark the corresponding random-guess baselines (25% for Point and 50% for Type and Layer).
We present FoldingAgent, an agentic framework for inferring explicit parametric folding programs directly from origami demonstration videos. Our framework leverages the reasoning power of a pre-trained Vision-Language Model (VLM) equipped with a suite of specialized tools that enable the agent to simulate geometric transitions, verify physical plausibility, retrieve and compare visual content, and evaluate its own predictions. To translate visual content into folding programs, we define a parametric space that consists of the paper's geometry and a set of parametric folding actions. Unlike models that predict static crease patterns, our agent operates sequentially and possesses the ability to re-plan its actions, effectively mitigating the compounding errors inherent in multi-step folding. Our approach takes a step toward closing the gap between human origami knowledge, which is primarily shared through unstructured visual demonstrations, and computational methods, which typically rely on structured, parametric representations such as a crease pattern or an executable parametric plan. We evaluate our approach on PurelandFold, a newly curated benchmark of diverse Pureland origami videos with ground-truth geometry and action labels. Our results demonstrate that by combining VLM reasoning with a set of specialized tools and physical simulation, we can successfully transform unstructured visual demonstrations into executable, physically plausible folding procedures.
Maya Moriya, Sigal Raab, Yael Vinker +1
Weizmann Institute of Science, Rehovot, Israel · MIT, Boston, USA
While generative AI has achieved remarkable success in solving problems with verifiable solutions, generating physical art that satisfies both strict geometric constraints and subjective visual aesthetics remains a challenge. This paper presents an approach to tackle these difficulties in the domain of computational origami, a mathematically rigid environment that grounds artistic design within the equations of flat foldability. We present COrigami, an end-to-end AI-driven pipeline that assists the design cycle by generating crease patterns from natural language. Our pipeline involves generating a semantic stick figure, computing a base packing, solving for a flat-foldable crease pattern, shaping the flat-folded crease pattern, and refining the generated model using reinforcement learning driven by an autonomous aesthetic evaluation loop. Our system acts as a highly effective collaborative assistant, generating structural starting points that human artists can further expand and shape. By integrating algorithmic optimisation with autonomous aesthetic critique, this work demonstrates how AI systems can satisfy multi-objective physical constraints to enable reliable, mathematically grounded co-creativity.
The physical world is not merely visual; it is governed by rigorous structural and procedural constraints. Yet, the evaluation of vision-language models (VLMs) remains heavily skewed toward perceptual realism, prioritizing the generation of visually plausible 3D layouts, shapes, and appearances. Current benchmarks rarely test whether models grasp the step-by-step processes and physical dependencies required to actually build these artifacts, a capability essential for automating design-to-construction pipelines. To address this, we introduce DreamHouse, a novel benchmark for physical generative reasoning: the capacity to synthesize artifacts that concurrently satisfy geometric, structural, constructability, and code-compliance constraints. We ground this benchmark in residential timber-frame construction, a domain with fully codified engineering standards and objectively verifiable correctness. We curate over 26,000 structures spanning 13 architectural styles, ach verified to construction-document standards (LOD 350) and develop a deterministic 10-test structural validation framework. Unlike static benchmarks that assess only final outputs, DreamHouse supports iterative agentic interaction. Models observe intermediate build states, generate construction actions, and receive structured environmental feedback, enabling a fine-grained evaluation of planning, structural reasoning, and self-correction. Extensive experiments with state-of-the-art VLMs reveal substantial capability gaps that are largely invisible on existing leaderboards. These findings establish physical validity as a critical evaluation axis orthogonal to visual realism, highlighting physical generative reasoning as a distinct and underdeveloped frontier in multimodal intelligence. Available at https://luluyuyuyang.github.io/dreamhouse