PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR
Authors: Esen K. Tütüncü, Qian Zhou, Frederik Brudy, George Fitzmaurice, Fraser Anderson
Organizations: Institute of Neurosciences of the University of Barcelona, Barcelona, Spain · Autodesk Research, San Francisco, USA · Autodesk Research, Toronto, Canada
Current AI writing tools, which rely on text prompts, poorly support the spatial and interactive nature of storytelling where ideas emerge from direct manipulation and play. We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props. A multi-agent AI pipeline interprets these actions into Intent Frames -structured narrative beats visualized as rearrangeable story marbles on a timeline. A large language model then transforms the user's assembled sequence into a final narrative. A user study (N=13) with writers from varying domains found that PlayWrite fosters a highly improvisational and playful process. Users treated the AI as a collaborative partner, using its unexpected responses to spark new ideas and overcome creative blocks. PlayWrite demonstrates an approach for co-creative systems that move beyond text to embrace direct manipulation and play as core interaction modalities.
Figures & tables
Figure 1. Overview of PlayWrite . Users create stories by (a) playing with characters and narrative props, (b) engaging in real-time dialogue with AI-driven characters, (c) assembling intent frames and marbles into narrative structure, and (d) exporting an assembled story. The system situates authoring in playful multimodal interaction, combining spatial manipulation and dialogue with AI. Overview of the system with a user wearing a mixed-reality headset standing at a tabletop surface. Small 3D characters and props such as Aladdin, a carpet, and some rocks are positioned on the table. A speech bubble and icons suggest that gestures and speech are observed by the system
Figure 2. PlayWrite Interface. (a) Grab to Talk: picking up a character triggers user’s voice input. (b) Drag to Move: repositioning the character’s transform handle for moving them in space with character animations. (c) Attach a Prop: objects can be linked to characters to contextualize actions. (d) AI Respond: the system generates dialogue and narrative continuation in response to user actions. (e) Marble Assembly: users arrange story marbles on a timeline, each representing an Intent Frame, to sequence and refine the story. Dialogue and character locations are replayed when a marble is selected, supporting reflection and revisiting. Different views of the PlayWrite interface layout. The main 3D playground shows characters and props. A dialogue window appears above characters when they speak. Below, a horizontal timeline displays circular “story marbles” that can be selected and rearranged.
Figure 3. System pipeline of PlayWrite . (1) Multi-agent observation collects user inputs (speech, grabbing, moving) and interprets them through Environmental, Social, and Narrator agents. (2) The Intent Frame Agent fuses these into higher-level features through temporal filtering, contextual prioritization, and cross-agent fusion. (3) Intent frames are assembled into marbles, which can be replayed, reordered, or exported. (4) Story generation produces a coherent story summary and screenplay from the assembled marbles. Flowchart-style diagram showing PlayWrite’s pipeline. On the left are user inputs (grabbing, moving, talking). In the center are three types of agents (Environmental, Social, Narrator). Arrows lead to Intent Frame generation and then to Story Assembly and Story Generation. Small icons of characters and objects appear alongside each stage.
Figure 4. Examples of semantically meaningful features captured during play and encoded into Intent Frames. (a) Zone of Interest: a character placed within a predefined hiding zone, and therefore, his action and speech will not be noticed by other characters. (b) Movement Trail: the movement of a character is tracked to capture directionality and pacing. (c) Character–Prop Interaction: a gun is attached to signal a threat. Other characters will react to seeing the gun in hand. (d) Character–Character Interaction: relative positioning and engagement between characters are recorded as two characters relaxing together in the pool. (e) Proximity: spatial closeness between the character and a magic carpet indicates plot progression. Together, these features provide structured cues that the system abstracts into Intent Frames for narrative interpretation. Exemplary interactions: sequence of frames showing a user arranging characters, triggering speech, assembling Story Marbles, and exporting a screenplay.
Figure 5. Recorded example workflow of Robin Hood extracted from a user’s play (P6, open-ended task). Green-labeled boxes show user speeches, and black-labeled boxes show AI responses demonstrating two conversation segments: (first row) Mary pleads with Robin Hood for help. The user, controlling Robin, confronts Lord Pemberton. (second row) The story develops as the user switches characters and overrides AI responses to introduce a plot twist. Complete dialogue and synopsis can be found in Appendix E and F. Exemplary interactions: sequence of frames showing a user arranging characters, triggering speech, assembling Story Marbles, and exporting a screenplay.
ID
Writing Background
Primary Writing Format
P1
Songwriting, interactive media storytelling, fantasy, science fiction, screenwriting, game narrative, poetry
Scripts/screenplays
P2
Poetry, journals, literary fiction
Short stories
P3
Fiction writing, poetry, literary fiction
Short stories
P4
Prose fiction, fantasy, science fiction, poetry
Short stories
P5
Science fiction writing, screenwriting, game narrative
Short stories
P6
Dungeon master, improv comedy, game narrative, interactive fiction
Improv comedy and D&D narratives
Table 1. Participant backgrounds, mapped to the anonymized IDs (P1–P13) used when reporting quotes and findings. Table listing each participant with demographics such as writing background and primary storytelling medium.
Figure 6. Three distinct scenes used in the user study. (a) Tutorial Scene: A simple environment depicting an interview between two characters with a desk zone and stage zone, including an Elvis impersonator, a hiring manager, and basic props. This scene was used to introduce participants to the system’s core interactions (grabbing, moving, speaking, attaching props). (b) Aladdin Scene: A desert-themed environment with pond and hide zones, featuring Aladdin, Genie, and Jafar alongside interactive props. Participants were encouraged to improvise and stage interactions using familiar characters. They were presented with an end goal to reach. (c) Robin Hood Scene: A castle environment with an open space zone, featuring Robin Hood, Mary, and Lord Pemberton, along with thematic props. Participants were instructed to use the characters and the props for an open-ended narrative exploration. Together, these three scenes provided varying levels of narrative structure, scaffolding participants from practice to open-ended improvisation. Three different scenes shown side-by-side. Each has unique characters, props, and a distinct environment theme. The images represent the pre-authored starting scenes used in the study.
Factor
Mean
SD
Expressiveness
6.35
0.77
Enjoyment
6.15
0.80
Immersion
5.85
1.41
Results Worth Effort
5.85
1.07
Exploration
5.38
1.43
Control
3.92
1.93
Table 2. Creativity Support Index (CSI) Results (N=13) Table showing Creative Support Index (CSI) subscale scores (e.g., Enjoyment, Exploration, Control).
Figure 7. Results of post-study questionnaires for: ( a ) Overall Experience, ( b ) Co-creative Partnership, and ( c ) Perceived Playfulness. Results of post-task questionnaires for: ( d ) Closed-Ended Task and ( e ) Open-Ended Task a) Bar chart showing participants’ responses to co-creativity and playfulness questions. The overall results are positive, with lower agreement in items related to precise control of the AI. b)Comparison bar chart across two storytelling modes: goal-oriented vs. open-ended. Each mode has multiple response bars showing different aspects of user experience; differences are visible in perceived exploration and challenge.
Figure 8. Action-level interaction analysis across participants. ( a ) Raw action beats for the Aladdin scenario, showing participant timelines with color-coded action events. ( b ) Stacked distribution of action types across participants in the Aladdin scenario. ( c ) Raw action beats for the Robin Hood scenario. ( d ) Stacked distribution of action types across participants in the Robin Hood scenario. Together, these visualizations highlight how participants combined speech and physical actions over time, and how relative reliance on each interaction type varied across individuals. The periods of user inactivity have been removed from the raw action beat visualizations (a, c) to focus on active interaction sequences. Stacked bar chart showing interaction types used by participants (e.g., grabbing, speech, moving). Colors differentiate interaction categories, and bars vary in length, showing differences in behavior and preference across participants.
Conversational AI agents are increasingly explored as creative partners, yet how conversation design shapes child-AI dialogue in co-creative settings remains underexplored. We present Tinker Tales, a tangible dialogue system for child-AI collaborative storytelling, in which educational frameworks (narrative development and social-emotional learning) are instantiated as conversation design, shaping how the agent engages children across four narrative stages. The system combines a physical storytelling board, NFC-embedded toys, and a mobile app mediating multimodal interaction through tangible manipulation and voice-based dialogue. We conducted a home-based user study with 10 children (ages 6-8) across two conversation design conditions varying in how the agent structured elaboration, with and without educational scaffolding. Our findings show that prompt framing shapes the form and consistency of children's narrative contributions, structuring how they participate in co-creative dialogue with AI.
Nayoung Choi, Jiseung Hong, Peace Cyebukayire +2
Department of Computer Science, Emory University · Department of Computer Science, Carnegie Mellon University · School of Nursing, Emory University
Large Language Models show promise for AI-assisted storytelling, yet current tools often generate predictable, unoriginal narratives. To address this limitation, we present NarrativeLoom, a multi-persona co-creative system grounded in Campbell's Blind Variation and Selective Retention theory. NarrativeLoom deploys specialized AI personas to generate diverse narrative options (blind variation), while users act as creative directors to select and refine them (selective retention). We designed a controlled study with 50 participants and found that stories co-authored with NarrativeLoom were not only perceived by users as more novel and diverse but were also objectively rated by experts as significantly better across all Torrance Test creativity dimensions: fluency, flexibility, originality, and elaboration. Stories are significantly longer with richer settings and more dialogue. Writing expertise emerged as a moderator: novices benefited more from structured scaffolding. This demonstrates the value of theory-informed co-creative systems and the importance of adapting them to varying user expertise.
Yuxi Ma, Yongqian Peng, Fengyuan Yang +5
Institute for Artificial Intelligence, Peking University Beijing, China · School of Psychological and Cognitive Sciences, Peking University Beijing, China · Also with the State Key Lab of General AI. +5
The topic of Co-creation, i.e., AI agents interacting with humans to generate outputs (e.g., art), has gained significant attention recently. However, most studies focus on adult-human interactions in a digital setting. This paper explores a novel ludic co-creation scenario involving children and Large Language Models (LLMs) interacting through a physical board game to create written stories. Our goal is to develop a multi-agent framework capable of producing high-quality narratives suitable for young players. At the core of our approach is an iterative Writer-Editor process in which one LLM generates stories while another evaluates them and provides feedback for refinement. Through a simulation study involving multiple LLMs, we show that this iterative interaction consistently improves the perceived quality of generated stories across successive loops. The results indicate that a small number of refinement steps may be sufficient to achieve high-quality outputs in interactive storytelling systems.
Arturo Valdivia, Paolo Burelli
Data Science Section · IT University of Copenhagen · Copenhagen, Denmark +1