FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows
Organizations: Patronus AI
Abstract
Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as Claude-Opus-5 and GPT-5.6-Sol on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing Qwen3.8-27B outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.
Figures & tables
| ID / Task type | Start-state creation method |
|---|---|
| 1. Pixel-perfect replication (open seeds) | Pull a page from the open-web whitelist. Capture full-page PNG at 1440 px (browser capture or SingleFile archive) and archive the URL, capture date, and license basis. Attach the one reusable instruction: “Replicate 1:1 in Figma with proper auto-layout.” |
| 2. Responsive / platform adaptation | Attach a per-target instruction template: desktop 375 px mobile; web tablet; print web; web app Android (Material). |
| 3. Theming with variables | Instruction template on any cleared seed: “Produce the dark/light variant implemented via Figma variables/modes—no manual per-node recolor.” |
| 4. Sketch-to-Figma | Start with a sourced hand-drawn sketch (existing hand-drawn-to-website datasets are also available). SME builds the hi-fi design from the photo. Both artifacts are owned via the contributor agreement. |
| 5. Flaw injection repair | Take an existing open-source Figma template and modify it to break a few things in the workflow. |
| 6. Edge-content injection resilience | Test whether a layout survives content it was never designed for—most screens are only ever tested against clean, friendly demo data. |
| Mind2Web | VideoGUI | |||||
| Model | GUI-Odyssey | AndroidControl | vp800 | vp1000 | directed | undirected |
| Random Baseline | 0.6 | 21.8 | 1.1 | 0.9 | 0.6 | 0.6 |
| Claude-Opus-5 | 47.3 | 87.3 | 100.0 | 75.3 | 71.3 | 40.7 |
| GPT-5.6-Sol | 44.0 | 83.2 | 91.3 | 69.7 | 77.7 | 29.6 |
| Qwen-3.6-35BA3B | 29.0 | 36.8 | 38.2 | 30.7 | 47.0 | 10.1 |
| Muse-Glimmer-30B | 29.3 | 59.3 | 64.7 | 65.3 | 64.7 | 28.7 |
| Benchmark | Base | Phase-based SFT | Length-based SFT |
|---|---|---|---|
| GUI-Odyssey | 44.0 | 53.7 | 40.0 |
| AndroidControl | 82.7 | 99.1 | 67.3 |
| Mind2Web vp800 | 69.3 | 70.7 | 69.3 |
| Mind2Web vp1000 | 65.3 | 68.7 | 70.0 |
| VideoGUI-undirected | 26.0 | 19.3 | 21.3 |
| VideoGUI-directed | 65.3 | 71.3 | 70.0 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Label | Definition | Share (%) |
|---|---|---|
| reference_gathering | Studying or collecting source material | 5.7 |
| setup_scaffolding | Establishing frames, artboards, grids, palettes, and styles | 7.7 |
| blocking_layout | Setting coarse structure, placement, and sizing | 16.1 |
| asset_sourcing | Searching for or importing images, icons, and fonts | 8.5 |
| content_entry | Typing real copy into the artifact | 8.9 |
| styling_typography | Making colour, type, and spacing decisions | 8.3 |
| Criterion | Status | Notes |
|---|---|---|
| 1. File Structure & Agent Navigation | ||
| Cover page with file name, version, and status | Nice to have | Agent does not use the cover page directly, but it helps human reviewers orient quickly before handing the file over. |
| Pages are logically organized (e.g. Cover / Components / Screens / Archive) | Pass/Fail | In accordance with Figma best practices |
| Pages are not overloaded and large files are split across multiple pages to avoid exceeding agent context limit | Pass/Fail | Source: Figma MCP docs specifically mention ‘If you call get_design_context on an entire page instead of a specific node, the response can easily exceed the 25,000-token context window.‘ and this is expected to reduce this single call load. |
| Each screen is a separate top-level frame with a descriptive name (e.g. ”Login Screen”, ”Home / Mobile”) | Pass/Fail | Source: Figma MCP documentation mentions agent uses get_metadata to navigate the file by reading frame names. For example, ”Frame 247” gives no orientation. |
Tool-Using Vision-Language Agents
Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.
See all 139 papers