Augmented reading systems aim to adapt text presentation to improve comprehension and task performance, yet existing approaches rely heavily on heuristics, opaque data-driven models, or repeated human involvement in the design loop. We propose framing augmented reading as a simulation-based optimization problem grounded in resource-rational models of human reading. These models instantiate a simulated reader that allocates limited cognitive resources, such as attention, memory, and time under task demands, enabling systematic evaluation of text user interfaces. We introduce two complementary optimization pipelines: an offline approach that explores design alternatives using simulated readers, and an online approach that personalizes reading interfaces in real time using ongoing interaction data. Together, this perspective enables adaptive, explainable, and scalable augmented reading design without relying solely on human testing.
Figures & tables
Figure 1. Resource-rational reading models enable principled optimization of text user interfaces across a wide design space. ( Left ) A reading agent interacts with text-based user interfaces by allocating visual attention over time, guided by perceptual input, memory state, task objectives, and an adaptive policy. ( Right ) A design space for augmented reading defined by user groups, text presentation designs, and task demands. The same underlying model can be instantiated to evaluate diverse reading scenarios, illustrated here with three examples: ( top ) a reader with restricted visual field (tunnel vision), ( middle ) reading supported by visually salient text cues (e.g., font size, type, and color), and ( bottom ) multitasking contexts such as reading while driving from an in-vehicle display. Together, the model enables principled comparison and optimization of text UI designs across users, tasks, and environments.
Figure 2. Optimization paradigms for augmented reading. (a) Human-in-the-loop optimization. Traditional text UI design relies on human readers to generate behavioral data and feedback, making optimization costly and difficult to scale. (b) Simulation-based offline optimization. A simulated reading agent represents human readers to evaluate and optimize candidate text UIs, enabling systematic exploration of the design space without repeated user studies. (c) Simulation-based online optimization. Human interaction data are used to initialize a personalized simulated agent that continues reading on the user’s behalf, supporting real-time evaluation and adaptive UI optimization during ongoing reading.
Figure 3. Gallery of simulation-based optimization for augmented reading. Three scenarios illustrate how a simulated reading agent predicts human-like reading behavior (red dots indicate fixations on text; blue dots indicate attention to the surrounding environment) and task performance (summarized by model-predicted metrics in orange boxes) under different contexts, and how these predictions guide UI optimization. Across (a) reading while walking on AR, (b) time-limited reading, and (c) visual search while driving, the model identifies bottlenecks in workload, comprehension, or safety. Targeted design interventions, such as increasing layout sparsity, reducing content load, or enhancing visual saliency lead to improved predicted reading behavior and task performance after optimization. (d) Individualized reading support demonstrates how reader-specific simulation reveals inefficient lexical access (e.g., slow reading speed and dense fixations), enabling personalized adaptations such as enhanced lexical saliency to improve predicted reading speed and comprehension.
We present a framework for evaluating adaptive personalization of educational reading materials with theory-grounded simulated learners. The system builds a learning-objective and knowledge-component ontology from open textbooks, curates it in a browser-based Ontology Atlas, labels textbook chunks with ontology entities, and generates aligned reading-assessment pairs. Simulated readers learn from passages through a Construction-Integration-inspired memory model with DIME-style reader factors, KREC-style misconception revision, and an open New Dale-Chall readability signal. Answers are produced by score-based option selection over the learner's explicit memory state, while BKT drives adaptation. Across three sampled subject ontologies and matched cohorts of 50 simulated learners per condition, adaptive reading significantly improved outcomes in computer science, yielded smaller positive but inconclusive gains in inorganic chemistry, and was neutral to slightly negative in general biology.
Ryan T. Woo, Anmol Rao, Aryan Keluskar +1
School of Computing and Augmented Intelligence, Arizona State University
This work presents the design, implementation, and evaluation of a system for generating personalized reading content using Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG). The proposed architecture consists of four modules: Input, RAG, Generation, and Judging and enables users to specify both a question and a target reading content complexity. RAG is employed to retrieve relevant information from the Internet, enriching and grounding the content produced by three modern LLMs: Meta LLaMA 4 Scout, LLaMA 3.1 8B Instant, and Google Gemma2 9B. Reading materials are generated using three prompting strategies (Chain-of-Thought, zero-shot, and few-shot), and the LLM-as-a-Judge module automatically evaluates answer quality and alignment with the desired readability level. Experimental results show that RAG consistently improves system performance across all models and prompting techniques, increasing relevance and particularly groundedness by up to 26-35 percentage points. Overall, the findings demonstrate that the RAG-augmented architecture effectively produces reading content tailored to user queries and desired textual complexity.
Sooyeon Kim, Piotr S. Maciąg
Institute of Computer Science, Warsaw University of Technology, Nowowiejska 15/19, 00-661 Warsaw, Poland
Automatic prompt optimization (APO) has shown strong effectiveness for text-based LLMs and LLM systems, but its application to audio-visual understanding systems requires suitable textual interfaces through which multimodal reasoning can be exposed to text-based optimizers. We investigate how the design of textual interfaces affects prompt optimization for multimodal reasoning, on the audio-visual question answering task. We compare three interface designs, including a direct inference system consisting of a single omni-modal large language model (OmniLLM), a captioner-reasoner system, and an agentic architecture in which a primary agent orchestrates subagent OmniLLMs. Experiments with Qwen3-Omni and Gemini show that direct prompt optimization provides model-dependent gains; the captioner-reasoner system and the agentic system expose more interpretable optimization interfaces but are harder to optimize, where error analysis of cases affected by optimization is conducted. Our study highlights textual interface design as an important factor in making prompt optimization effective for omni-modal reasoning systems. Our implementation details are specified in GitHub repo.
Yi Wang, Tianzi Wang, Tong Lei +2
School of Informatics, University of Edinburgh, UK · Department of Systems Engineering and Engineering Management, Chinese University of Hong Kong, Hong Kong SAR · Key Laboratory of Modern Acoustics, Institute of Acoustics, Nanjing University, China