Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as Claude-Opus-5 and GPT-5.6-Sol on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing Qwen3.8-27B outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.
Figures & tables
Figure 1: A sample FigmaTrace trajectory excerpt taken from session t1-112 , task involving replicating the Nuxt Website.
Figure 2: A comprehensive set of relevant skills that FigmaTrace covers.
ID / Task type
Start-state creation method
1. Pixel-perfect replication (open seeds)
Pull a page from the open-web whitelist. Capture full-page PNG at 1440 px (browser capture or SingleFile archive) and archive the URL, capture date, and license basis. Attach the one reusable instruction: “Replicate 1:1 in Figma with proper auto-layout.”
2. Responsive / platform adaptation
Attach a per-target instruction template: desktop → 375 px mobile; web → tablet; print → web; web app → Android (Material).
3. Theming with variables
Instruction template on any cleared seed: “Produce the dark/light variant implemented via Figma variables/modes—no manual per-node recolor.”
4. Sketch-to-Figma
Start with a sourced hand-drawn sketch (existing hand-drawn-to-website datasets are also available). SME builds the hi-fi design from the photo. Both artifacts are owned via the contributor agreement.
5. Flaw injection → repair
Take an existing open-source Figma template and modify it to break a few things in the workflow.
6. Edge-content injection resilience
Test whether a layout survives content it was never designed for—most screens are only ever tested against clean, friendly demo data.
Table 1: Taxonomy of Figma design task categories and their start-state creation methods.
Figure 3: Combined-corpus composition in share percentage. Actions follow the Playwright MCP toolset, counted over all validated tool calls. Phases come from the 11-way taxonomy as described in subsection A.1 . Bars are scaled within each panel and shares are rounded. The skill mix is listed out separately in Figure 4 .
Mind2Web
VideoGUI
Model
GUI-Odyssey
AndroidControl
vp800
vp1000
directed
undirected
Random Baseline
0.6
21.8
1.1
0.9
0.6
0.6
Claude-Opus-5
47.3
87.3
100.0
75.3
71.3
40.7
GPT-5.6-Sol
44.0
83.2
91.3
69.7
77.7
29.6
Qwen-3.6-35BA3B
29.0
36.8
38.2
30.7
47.0
10.1
Muse-Glimmer-30B
29.3
59.3
64.7
65.3
64.7
28.7
Table 2: Step-wise accuracy (%) across OOD GUI agent benchmarks. For VideoGUI, directed refers to step-wise instruction provision whereas undirected is open-ended navigation. Mind2Web tests are run in two different image viewport (vp) configurations. Bold indicates the best open model in each column and Underline represents the best closed model. ∗ Muse-Glimmer-30B was evaluated similarly to the official evaluation with multiple zoom and crops averaged.
Benchmark
Base
Phase-based SFT
Length-based SFT
GUI-Odyssey
44.0
53.7
40.0
AndroidControl
82.7
99.1
67.3
Mind2Web vp800
69.3
70.7
69.3
Mind2Web vp1000
65.3
68.7
70.0
VideoGUI-undirected
26.0
19.3
21.3
VideoGUI-directed
65.3
71.3
70.0
Table 3: Comparison of phase-aware SFT against the base Qwen3.8-27B model and a maximum-length based SFT baseline across six GUI agent benchmarks. Best result per row in bold .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Skill mix by share percentage across all skill-labeled trajectories in FigmaTrace .
Label
Definition
Share (%)
reference_gathering
Studying or collecting source material
5.7
setup_scaffolding
Establishing frames, artboards, grids, palettes, and styles
7.7
blocking_layout
Setting coarse structure, placement, and sizing
16.1
asset_sourcing
Searching for or importing images, icons, and fonts
8.5
content_entry
Typing real copy into the artifact
8.9
styling_typography
Making colour, type, and spacing decisions
8.3
Appendix
Table 4: The closed phase vocabulary. Labels are ordered by their typical position in the workflow rather than by frequency.
Criterion
Status
Notes
1. File Structure & Agent Navigation
Cover page with file name, version, and status
Nice to have
Agent does not use the cover page directly, but it helps human reviewers orient quickly before handing the file over.
Pages are not overloaded and large files are split across multiple pages to avoid exceeding agent context limit
Pass/Fail
Source: Figma MCP docs specifically mention ‘If you call get_design_context on an entire page instead of a specific node, the response can easily exceed the 25,000-token context window.‘ and this is expected to reduce this single call load.
Each screen is a separate top-level frame with a descriptive name (e.g. ”Login Screen”, ”Home / Mobile”)
Pass/Fail
Source: Figma MCP documentation mentions agent uses get_metadata to navigate the file by reading frame names. For example, ”Frame 247” gives no orientation.
School of Microelectronics, Southern University of Science and Technology (SUSTech), Shenzhen, China · Zhongguancun Academy, Beijing, China · School of Marxism, Tsinghua University, Beijing, China