Although HCI research papers offer valuable design insights, designers often struggle to apply them in design workflows due to difficulties in finding relevant literature, understanding technical jargon, the lack of contextualization, and limited actionability. To address these challenges, we present ReFinE, a Figma plugin that supports real-time design iteration by surfacing contextualized insights from research papers. ReFinE identifies and synthesizes design implications from HCI literature relevant to the mockup's design context, and tailors this research evidence to a specific design mockup by providing actionable visual guidance on how to update the mockup. To assess the system's effectiveness, we conducted a technical evaluation and a user study. Results show that ReFinE effectively synthesizes and contextualizes design implications, reducing cognitive load and improving designers' ability to integrate research evidence into UI mockups. This work contributes to bridging the gap between research and design practice by presenting a tool for embedding scholarly insights into the UI design process.
Figures & tables
Figure 1. An overview of the ReFinE -aided design workflow. Based on a user’s selected UI mockup on the Figma canvas, the system generates \footnotesize{1}⃝ clusters of research insight informed by HCI research papers. Designers can examine \footnotesize{2}⃝ each cluster to understand the synthesized design implications before reviewing \footnotesize{3}⃝ action items and their visualizations, which project how potential design edits could be made to the interface. Designers can use these to perform designerly iteration, modifying or adapting the proposed changes at their own discretion. They can also explore the \footnotesize{4}⃝ source papers that informed each cluster.
Figure 2. Pipeline for generating the components provided by ReFinE
Figure 3. Contents of the cluster and their formulation
Figure 4. An example of mockup reconstruction in our pipeline. Although the reconstructed HTML may not perfectly match the original mockup, it faithfully mirrors the components in the original mockup and allows for visualization of action items.
Figure 5. An example of an action item and its visualization. With the reconstructed HTML, ReFinE selects the appropriate screen(s) from the design mockup and visualizes how each action item can be applied. Users can toggle between the before and after visualization, assisting designers in grasping the differences. Users can also bookmark each action item.
Figure 6. Design outputs from iterating two example mockups with ReFinE -generated action items. For each example, we present the extracted design context, an example of retrieved design implications, and action items used for the iteration. For conciseness, a single screen pair (before/after) is shown for each mockup, and edits are highlighted with a yellow background.
System step
Latency (seconds)
Eliciting design context dimensions
3.167±1.019
Mockup reconstruction*
10.794±3.021
Retrieval & clustering
2.004±0.400
Generating translated insights*
2.982±0.326
Generating action items
4.435±0.480
Visualizing action item
2.913±1.215
Table 1. Overall system latency analysis (metrics are computed per task ± stdev, *: per unit; otherwise per design)
Figure 7. Perceived workload, communication quality, and the design output metrics from our user study. Participants in the ReFinE condition reported significantly reduced workload compared to the baseline condition, while finding the communicated design insights to be significantly improved. Additionally, participants made significantly more edits interacting with ReFinE , while reaching iteration more quickly. Error bars indicate standard errors. (**: p<.01 , ***: p<.001 ; †: reverse item)
Figure 8. P4’s example of implementing an action item. After recognizing its potential usefulness, P4 creatively adapted the action item to a different screen, redesigning it with alternative UI components.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Visual similarity
Latency (seconds)
Gemini 2.0 Flash ( Google, 2024b )
0.7875±0.1437
10.794±3.021
Claude 3.5 Sonnet ( Anthropic, 2024 )
0.7963±0.1590
17.006±4.914
GPT-4o ( OpenAI, 2024 )
0.7852±0.1330
11.060±3.382
Appendix
Table 2. Comparison of VLMs for HTML reconstruction using image inputs (metrics are computed per-screen ± stdev)
Input modality
Visual similarity
Latency (seconds)
Image
0.7875±0.1437
10.794±3.021
JSON
0.7661±0.1314
15.432±7.047
Image + JSON
0.7711±0.1411
10.630±2.427
Appendix
Table 3. Comparison of different input modalities on HTML reconstruction (base model is Gemini 2.0 Flash, metrics are computed per screen ± stdev)
Method
Accuracy
Latency (seconds)
Complete HTML regeneration
96.0%
10.248±3.488
Edit-only generation
95.3%
2.913±1.215
Appendix
Table 4. Comparison of methods for generating visualizations of action items applied to designs (metrics are computed per action item ± stdev)
Relevance rating
Design implication
Action item
Annotator’s explanation
5 - Highly relevant
Reduce user burden by minimizing text input in form-heavy tasks
Implement a one-click autofill feature that uses stored preferences for all checkout fields
Directly addresses and solves the burden
4 - Substantially relevant
Make it easier for users to navigate dense information in reports
Add a ‘Back to Top’ button on the settings screen
Helps with navigation, but not specific to navigating documents
3 - Moderately relevant
Striking the balance between insufficient and overwhelming transparency will enable users to trust the system
Add a ‘Report’ button to the comment section to enhance user feedback and platform moderation
May support trust, but unclear link to transparency or balance
2 - Slightly relevant
Allow users to control the visibility of their personal data in shared platforms
Let users toggle switches in the profile settings to show/hide recent activity
Some connection to visibility, but lacks clarity on shared context
1 - Not relevant
Help users build trust in financial transactions through transparency
Display an autoplay video on the app with product promotions
Unrelated to trust or transparency
Appendix
Table 5. Examples of relevance ratings assigned by annotators when evaluating the connection between design implications and action items This table illustrates examples of relevance ratings from 1 (Not relevant) to 5 (Highly relevant), assigned by annotators assessing how well action items align with design implications. Each row includes a rating, a design implication, a corresponding action item, and a brief explanation justifying the assigned relevance.
Figure 9. Prototype mockups used in our user study
Human-computer interaction (HCI) is in the middle of a transition: large language models can now generate functional user interfaces (UIs) on demand from natural-language task descriptions. A user explains what they are trying to accomplish, and the system materializes a working interface to support it. This capability already exists in systems such as Claude and ChatGPT and continues to grow in fidelity as the underlying models improve. The next step along this trajectory is to move from interfaces that are merely generated to interfaces that are generated well. We propose a framework in which the dialogue between user and artificial intelligence (AI) becomes a Space to Think: a shared, structured cognitive workspace in which task decomposition produces an on-demand user interface as an extension of the user's thinking rather than as a separate artifact. Within this paradigm, classical HCI design knowledge (Nielsen's heuristics, Norman's affordance prescriptions, Web Content Accessibility Guidelines (WCAG) success criteria, cognitive-load constraints, and mixed-initiative principles) is encoded as skills: machine-readable skill.md files that the generating agent loads at runtime as software engineering tools. Skills turn HCI design knowledge into declarative, inspectable, version-controlled, and editable artifacts owned by the HCI community itself so that accessibility, learnability, and consistency become properties of a generative process rather than properties of a finished product. We outline a research agenda depicting a future where the HCI field transitions from today's design and knowledge heuristic checklist towards a future where the craft becomes machine-readable, executable, and open.
Nathan Conklin, Miranda Capra, Chris North
Virginia Tech Blacksburg, Virginia, USA · Independent Consultant Chapel Hill, North Carolina, USA
Generative UI tools promise to democratize UI design by turning natural language descriptions into complete interfaces. Alongside the interface, these tools generate user-facing design rationales that explain their layout, accessibility, and design choices. However, it remains unclear whether these stated rationales are actually reflected in the interfaces they produce. We call this disconnect ``Design Theater'': plausible and confident design rationales that have little relationship to the actual implementation. To study this phenomenon, we introduce a benchmark and three metrics for measuring Design Theater. The benchmark includes 24 UI generation tasks spanning structural, styling, and functional design requirements. Using this benchmark, we evaluate 120 interfaces created by five generative UI tools. On average, over 25% of user-facing design rationales are not implemented in the generated interface, and the implementation failure increases to 34% for functional requirements. Tools recognize roughly half of the UX principles embedded in prompts (mean = 0.54), with four of five tools implementing 6% or fewer functional principles. We also measure interface similarity across tools and find convergence in visual appearance and layout organization, with greater variation in color choices. Overall, we contribute: 1) the concept of Design Theater; 2) a benchmark with metrics for assessing whether the stated reasoning of generative UI tools is reflected in their implementations; 3) and findings from a systematic evaluation of these tools. We discuss what these findings mean for the design and evaluation of generative UI tools.
Kashif Imteyaz, Kaif Imteyaz, Nakul Rajpal +3
Northeastern University, United States · Jamia Hamdard, India · Saarland University, Germany +2
While large language models (LLMs) and coding agents are often applied to user interface (UI) development, developers find it difficult to reliably assess their proficiency in visual and interaction design. Existing evaluations either rely on human experts, who can accurately assess usability by testing critical flows but are slow and costly, or on automated judges, which are scalable but less accurate and opaque. We present FlowEval, a reference-based framework that measures whether a generated UI supports realistic interaction flows by comparing navigation traces from real websites to traces from generated analogs using reference-based similarity metrics (e.g., dynamic time warping). In a small-scale study with expert UI evaluators, we show that reference-based metrics strongly correlate with human judgments, suggesting that they can provide scalable yet trustworthy evaluation for UI generation systems.