Evaluating user experience (UX) on live websites through user testing is expensive, subjective, and difficult to scale. LLM agents offer a promising route to automating UX testing by simulating realistic user behaviour. However, existing simulation approaches typically lack multimodality and require time-consuming manual review to extract actionable insights. We formalise UX improvement recommendation from simulation data as a structured natural language generation and ranking problem, and establish an evaluation protocol using expert annotation and LLM-as-a-Judge. We present AMUSER, a multimodal framework which simulates user behaviour and automatically generates prioritised UX improvement recommendations from resulting data. We evaluate AMUSER on commercial websites and show that its recommendations substantially outperform those from text-only simulation (NDCG@3 = 0.758 versus 0.359) at an 89% lower simulation cost. Our results suggest an asymmetric role of multimodality: visual access during simulation improves recommendations through richer traces, while providing visual inputs during recommendation generation can modestly degrade quality. We also discuss practical deployment lessons from applying AMUSER to commercial websites.
Figures & tables
Figure 1: Selected steps from an AMUSER simulation (young professional persona, searching for size 12 cotton trousers on sale). The agent navigates the site, producing thoughts and emotion labels at each step; the resulting trace D is converted into prioritised recommendations R (abridged recommendations shown for brevity).
Figure 2: Overview of the AMUSER framework. A persona analysis module conditions a simulation agent, which produces multimodal traces; a recommendation module generates prioritised product, UI, and UX recommendations.
Figure 3: Performance of baselines for medium and high priority recommendations by category relative to AMUSER. AMUSER outperforms across all categories.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Simulation Method
NDCG@3
NDCG
All
Product
UI
UX
Homepage HTML-only
0.402
0.515
0.269
0.333
0.173
0.299
UXAgent
0.503
0.628
0.395
0.367
0.398
0.419
AMUSER (ours)
0.846
0.922
0.639
0.793
0.447
0.682
Appendix
Table 3: GPT-5.5 robustness check on main results. Findings closely match the original GPT-5.4 results.
Ablation
NDCG@3
NDCG
All
Product
UI
UX
AMUSER
0.878
0.937
0.663
0.812
0.500
0.676
w/o images
+0.052
+0.026
+0.047
+0.061
+0.060
+0.022
w/o actions & thoughts
− 0.039
− 0.010
− 0.053
+0.037
− 0.090
− 0.083
Appendix
Table 4: GPT-5.5 robustness check on a 33% sample of ablations. Deltas shown relative to AMUSER.
Figure 4: An example of an AMUSER recommendation implemented on a retail category page. The “Show numeric sizes alongside letters” recommendation surfaced in Figure 1 was implemented via JavaScript injection (due to lack of access to the site’s codebase) using Claude Code with Opus 4.8 in a single shot. This provides a concrete example that AMUSER’s prioritised recommendations R can be actioned by site owners. Full recommendation title and description in Appendix C.3.1 .
AMUSER
AMUSER w/o images
UXAgent (non-multimodal)
Compact card layout*
Compact card layout*
Maintain current clarity
Emphasise distance visually*
Maintain map-list sync*
Enhance visual hierarchy
Appendix
Table 5: Illustrative (abridged) UI recommendations for a “find a nearby bank branch” simulation where multimodality was necessary in friction identification. Starred (*) recommendations require multimodal simulation to identify. In its recommendation description, UXAgent claims distance is ’prominently shown’; however, AMUSER’s trace shows it was below the fold. Text-only simulation has no viewport to detect this but the judge correctly flags UXAgent’s recommendation as incorrect. Recall that in the "w/o images" treatment, simulations are generated with multimodality but these images are not provided to the recommendation module at generation time. The user actions and frustrated thoughts captured by multimodal simulation persist in the text trace which explains why the recommendation module captures these frictions, without being shown image data.
Figure 5: Mean usefulness of recommendations as a function of simulation length (total steps), fit with a linear regression per category. Longer simulations are associated with higher-rated recommendations across all treatments and categories.
Evaluating web usability typically requires time-consuming user studies and expert reviews, which often limits iteration speed during product development, especially for small teams and agile workflows. We present Avenir-UX, a user-experience evaluation agent that simulates user behavior on websites and produces standardized usability. Unlike traditional tools that rely on DOM parsing, Avenir-UX grounds actions and observations, enabling it to interact with real web pages end-to-end while maintaining a coherent trace of the user journey. Building on Avenir-Web, our system pairs this robust interaction with simulated user behavior profiles and a structured evaluation protocol that integrates the System Usability Scale (SUS), step-wise Single Ease Questions (SEQ), and concurrent Think Aloud. Subsequently, a comprehensive User Experience (UX) report will be generated. We discuss the architecture of Avenir-UX and illustrate how its multimodal grounding improves robustness for web-based interaction and UX evaluation scenarios, paving the way for a new era of continuous, scalable, and data-driven usability testing that empowers every developer to build web interfaces that are usable. Code is available at: https://github.com/Onflow-AI/Avenir-UX
Wee Joe Tan, Zi Rui Lucas Lim, Shashank Durgad +2
University College London London, United Kingdom Work done at UCL Nexus Labs.
User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs' ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.
Large language models (LLMs) are increasingly deployed as UX judges that inspect interfaces, diagnose usability problems, and propose repairs. Yet no controlled benchmark measures whether the resulting critiques are reliable and actionable across heterogeneous product surfaces. We introduce UXBench, a benchmark for evaluating LLMs as interaction-grounded UX judges. UXBench comprises local-first runnable web fixtures spanning ten product-surface families, paired with coverage-gated browser exploration that forces models to collect interaction evidence before reporting. Each judge model produces a structured UX report over seven rubric dimensions; report quality is measured by whether a fixed downstream repair agent can improve the interface based on the critique. We evaluate eight frontier models under both an automated repair-lift protocol and a blind human validation study. Results show that UX judging is neither saturated nor one dimensional: models differ meaningfully in report actionability, exhibit distinct rubric-level repair signatures, vary in fixture-level reliability, and trade leadership across surface categories
Wenjie Wang, Yue Huang, Zipeng Ling +11
University of Notre Dame · University of Pennsylvania · University of Rochester +4