MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification
Organizations: Stanford University · Figma
Abstract
Large language models can generate motion graphics from static images and natural language requests, but it remains difficult to evaluate how well models follow requests that combine spatial, appearance, and temporal requirements. We introduce MotionGraphicsBench, an expert-curated benchmark of 100 instances for animating an input static SVG with a motion edit prompt. By supplying all required visual elements, our editing task reflects a designer-like workflow and isolates motion generation from graphic generation. Derived from real-world Lottie motion designs, our benchmark contains diverse combinations of spatio-temporal animation behaviors. Each benchmark instance pairs its model input with a deterministic, human-readable verification program that accepts different valid implementations and reports which prompt requirements pass or fail, rather than comparing against a reference animation or using a VLM judge. We construct these programs with a multi-agent pipeline using human-labeled animations and synthesized test cases. On a balanced test set of 180 animations, our programs achieve 93.3% raw agreement with human judgments, outperforming frontier VLM judges by 4.4%-12.8% while achieving at least 90% recall for both human approvals and rejections. Across 47 configurations from 32 LLMs and 12 providers, pass@1 ranges from 5.0% to 68.0%. Across 15 models at their highest reasoning effort, additional attempts improve every model, but pass@32 still ranges from 38.0% to 95.0%. Our analysis of verification programs reveals that generated animations often fail to match reference poses, follow requested motion trajectories, or coordinate related events. We fine-tune Qwen3.8-27B with group relative policy optimization (GRPO) using our verification programs as rewards, improving pass@1 from 30.31% to 52.19% on held-out instances. Our project page is at https://motion-graphics-bench.github.io/.
Figures & tables
| Requirement family | Types | Instances | Highest-coverage types (instances) |
|---|---|---|---|
| Spatial structure | 8 | 100 | Geometric state constraints (99); part attachment and group integrity (78); translation trajectories (72) |
| Temporal sequencing | 5 | 100 | Temporal ordering (100); temporal coordination and phase (93); animation speed and easing (93) |
| Appearance and visibility | 4 | 65 | Visibility and opacity (53); occlusion and visible-region containment (30); color, fill, and stroke appearance (13) |
| Method | Raw Agreement | Approval Recall | Rejection Recall |
|---|---|---|---|
| Opus 5 | 80.6% | 100.0% | 61.1% |
| Opus 5.5 | 88.9% | 96.7% | 81.1% |
| GPT-6 Astra | 83.3% | 82.2% | 84.4% |
| GPT-6.1 Sol | 82.8% | 83.3% | 82.2% |
| Our verification programs | 93.3% | 90.0% | 96.7% |
| Construction condition | Raw agreement | Approval recall | Rejection recall |
|---|---|---|---|
| Without human-labeled examples or synthesized test cases | 78.2% | 63.7% | 92.6% |
| Without human-labeled examples | 82.4% | 73.3% | 91.5% |
| Full construction | 93.3% | 90.0% | 96.7% |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Family and requirement type | Definition | Distinct instances |
|---|---|---|
| Spatial structure | ||
| Geometric state constraints | Required pose, position, orientation, dimensions, or contour at a specified phase or interval. | 99 |
| Part attachment and group integrity | Parts remain attached or inherit a common group motion or deformation. | 78 |
| Translation trajectories | Positional movement with a requested direction, path, or distance. | 72 |
| Transform origins and anchors | A requested rotation or scaling center, or a point or edge that remains anchored. | 71 |
| Relative position, alignment, and contact | Required alignment, spacing, contact, separation, or geometric containment relative to another element. | 66 |
| Set | Human labels approve/reject | Animations/program min/median/max | Programs with animations | Agreement | Approval recall | Rejection recall |
|---|---|---|---|---|---|---|
| Test set | 90/90 | 1/1/4 | 27 | 168/180 (93.3%) | 81/90 (90.0%) | 87/90 (96.7%) |
| All held-out | 90/274 | 1/4/4 | 97 | 334/364 (91.8%) | 81/90 (90.0%) | 253/274 (92.3%) |
| Misjudged requirement | Judgments |
|---|---|
| Part connections, grouping, and pivots | 43 |
| Motion direction, amount, or count | 35 |
| Shape and appearance | 34 |
| Timing and coordination | 29 |
| Spatial placement and reference poses | 26 |
| Benchmark instance | Generator | Human / program | Reviewed discrepancy |
|---|---|---|---|
| Smiling Face With Sunglasses (10798981) | GPT-5.5 | Reject / Approve | The smile does not follow the face bounce, and the lens-shine motion is incomplete; no check detects either failure. |
| Restaurant (11805404) | Gemini 3.1 Pro | Reject / Approve | The knife overlaps the fork and rotates about the wrong pivot; no check detects the required relative motion. |
| Finger Heart Symbol (8391483) | Gemini 3.1 Pro | Reject / Approve | The hand does not squeeze or dip, and the fingers do not rotate in opposing directions; no check detects these failures. |
| Pagoda (11510987) | Gemini 3.1 Pro | Approve / Reject | The program rejects the human-approved animation on its assembly-order check. |
| Loader (11640583) | Sonnet 5 | Approve / Reject | The centered-outline and consistent-stroke-thickness checks reject the human-approved rendering. |
| Password Protected (11971635) | Gemini 3.1 Pro | Approve / Reject | The synchronized four-tip wiggle check rejects the human-approved timing. |
| Configuration | Human (%) | Program (%) | Diff. (pp) |
|---|---|---|---|
| Sonnet 5 (xhigh) | 25.5 | 27.6 | +2.0 |
| Opus 4.8 (xhigh) | 30.0 | 38.0 | +8.0 |
| Opus 5 (max) | 55.0 | 61.0 | +6.0 |
| GPT-5.5 (xhigh) | 37.0 | 42.0 | +5.0 |
| GPT-5.6 Luna (max) | 23.0 | 31.0 | +8.0 |
| GPT-5.6 Sol (max) | 44.4 | 51.5 | +7.1 |
| Construction condition | Run | Agreement | Approval recall | Rejection recall |
|---|---|---|---|---|
| Without human-labeled examples or synthesized test cases | 1 | 80.56% | 67.78% | 93.33% |
| 2 | 76.11% | 60.00% | 92.22% | |
| 3 | 77.78% | 63.33% | 92.22% | |
| Without human-labeled examples | 1 | 82.78% | 73.33% | 92.22% |
| 2 | 83.33% | 76.67% | 90.00% | |
| 3 | 81.11% | 70.00% | 92.22% |
| Program version | Mean agreement |
|---|---|
| Iteration 1 | 61.79% |
| Iteration 2 | 88.32% |
| Iteration 3 | 91.28% |
| Iteration 4 | 92.02% |
| Iterations 5–7 | 93.32% |
| Final verification programs | 93.33% |
| Policy | Mean fraction of verification checks passed | Pass@1 | Pass@32 |
|---|---|---|---|
| Base model | 67.05% | 30.31% | 60.0% |
| After 41 updates | 84.22% | 52.19% | 100.0% |
| Model / version | Provider | Effort | Token cap | |
|---|---|---|---|---|
| GPT-6 Astra | OpenAI | high | 1 | 128,000 |
| GPT-6 Astra | OpenAI | max ∗ | 32 | 128,000 |
| GPT-5.6 Sol | OpenAI | high C | 1 | 100,000 |
| GPT-5.6 Sol | OpenAI | xhigh C | 1 | 100,000 |
| GPT-5.6 Sol | OpenAI | max ∗,C | 32 | 128,000 |
| GPT-5.6 Terra | OpenAI | high C | 1 | 100,000 |
| Models | Temperature | Top- | Top- |
|---|---|---|---|
| Gemma 4 31B IT; Muse Glimmer 30B | 1.0 | 0.95 | 64 |
| Qwen 3.8; Qwen 3.8 Flash-Next | 1.0 | 0.95 | 20 |
| Qwen 3.6 27B | 0.6 | 0.95 | 20 |
| GLM 5.2; GLM 5.3; Inkling; Inkling Small | 1.0 | 1.0 | – |
| GLM 5.3 Flash; DeepSeek V4 Pro; Kimi K3; MiniMax M3; Nemotron 3 Ultra | 1.0 | 0.95 | – |
| DeepSeek V4 Flash | 1.0 | – | – |
| Configuration | Pass@1 [95% CI] | Configuration | Pass@1 [95% CI] |
|---|---|---|---|
| Fable 5.1 (max) | 68.0 [59.0, 76.0] | GPT-6 Astra (max) | 67.0 [58.0, 76.0] |
| GPT-6 Astra (high) | 60.0 [50.0, 69.0] | GPT-5.6 Sol (max) | 52.0 [42.0, 62.0] |
| Opus 5 (max) | 49.0 [39.0, 58.0] | GPT-5.6 Sol (xhigh) | 48.0 [38.0, 58.0] |
| GPT-5.6 Terra (max) | 46.0 [36.0, 56.0] | Opus 5 (xhigh) | 45.0 [36.0, 54.0] |
| GPT-5.5 (xhigh) | 43.0 [33.0, 53.0] | Muse Spark 1.3 (max) | 41.0 [32.0, 50.0] |
| Opus 4.8 (xhigh) | 41.0 [31.0, 50.0] | Grok 4.6 (xhigh) | 41.0 [31.0, 51.0] |
| Configuration | Pass@32 [95% CI] | Configuration | Pass@32 [95% CI] |
|---|---|---|---|
| Fable 5.1 (max) | 95.0 [90.0, 99.0] | GPT-6 Astra (max) | 94.0 [89.0, 98.0] |
| GPT-5.6 Sol (max) | 86.0 [79.0, 92.0] | Opus 5 (max) | 86.0 [79.0, 92.0] |
| Muse Spark 1.3 (max) | 80.0 [72.0, 87.0] | Grok 4.6 (xhigh) | 75.0 [66.0, 83.0] |
| Qwen 3.8 (xhigh) | 82.0 [74.0, 89.0] | GLM 5.3 (max) | 83.0 [76.0, 90.0] |
| Gemini 3.8 Flash (high) | 68.0 [59.0, 77.0] | DeepSeek V4.1 Flash (max) | 80.0 [72.0, 87.0] |
| Kimi K3 (max) | 76.0 [67.0, 84.0] | DeepSeek V4 Pro (max) | 66.0 [57.0, 75.0] |
| Model / release comparison | Effort | Before | After | Change |
|---|---|---|---|---|
| Within-model effort changes | ||||
| GPT-5.6 Sol | high to xhigh | 35.0 | 48.0 | +13.0 |
| GPT-5.6 Terra | high to xhigh | 21.0 | 29.0 | +8.0 |
| GPT-5.6 Luna | high to xhigh | 25.0 | 32.0 | +7.0 |
| Opus 5 | high to xhigh | 36.0 | 45.0 | +9.0 |
| DeepSeek V4 Flash | high to max | 31.0 | 29.0 | -2.0 |
| Failure mode | Condition violated |
|---|---|
| Reference-pose mismatch | A requested starting, ending, return, or resting pose does not match its reference. |
| Motion trajectory mismatch | A translation or rotation follows the wrong path, direction, excursion, angle, or destination. |
| Event coordination mismatch | Related actions have the wrong order, phase, overlap, or coordination. |
| Linked-part or joint mismatch | Carried or attached parts, rigid groups, or named joints do not preserve their requested relationships. |
| Spatial relation or contact mismatch | Positions, alignment, separation, contact, or geometric containment violate a requested relationship. |
| Transform-origin mismatch | A rotation, scale, or deformation uses the wrong requested center, hinge, pivot, or anchor. |
| Failure mode | Instances | Failure rate (coverage) |
|---|---|---|
| Reference-pose mismatch | 98 | 41.50 (94.28) |
| Motion trajectory mismatch | 95 | 38.75 (88.63) |
| Event coordination mismatch | 95 | 32.41 (93.04) |
| Linked-part or joint mismatch | 77 | 39.51 (90.53) |
| Spatial relation or contact mismatch | 51 | 40.47 (94.10) |
| Transform-origin mismatch | 55 | 36.76 (92.45) |
| Failure mode | First model | Second model |
|---|---|---|
| Fable 5.1 (68.0) / GPT-6 Astra (67.0) | ||
| Reference-pose mismatch | 3.9 (84.8) | 14.5 (91.4) |
| Event coordination mismatch | 4.3 (85.4) | 2.1 (86.3) |
| Grok 4.6 (41.0) / Muse Spark 1.3 (41.0) | ||
| Reference-pose mismatch | 27.2 (84.7) | 20.7 (83.6) |
| Shape or scale mismatch | 2.5 (75.7) | 4.1 (78.9) |
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 335 papers