cs.HCMar 2, 2026
SavePlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR
Organizations: Institute of Neurosciences of the University of Barcelona, Barcelona, Spain · Autodesk Research, San Francisco, USA · Autodesk Research, Toronto, Canada
Abstract
Current AI writing tools, which rely on text prompts, poorly support the spatial and interactive nature of storytelling where ideas emerge from direct manipulation and play. We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props. A multi-agent AI pipeline interprets these actions into Intent Frames -structured narrative beats visualized as rearrangeable story marbles on a timeline. A large language model then transforms the user's assembled sequence into a final narrative. A user study (N=13) with writers from varying domains found that PlayWrite fosters a highly improvisational and playful process. Users treated the AI as a collaborative partner, using its unexpected responses to spark new ideas and overcome creative blocks. PlayWrite demonstrates an approach for co-creative systems that move beyond text to embrace direct manipulation and play as core interaction modalities.
Figures & tables
Figure 1. Overview of PlayWrite . Users create stories by (a) playing with characters and narrative props, (b) engaging in real-time dialogue with AI-driven characters, (c) assembling intent frames and marbles into narrative structure, and (d) exporting an assembled story. The system situates authoring in playful multimodal interaction, combining spatial manipulation and dialogue with AI. Overview of the system with a user wearing a mixed-reality headset standing at a tabletop surface. Small 3D characters and props such as Aladdin, a carpet, and some rocks are positioned on the table. A speech bubble and icons suggest that gestures and speech are observed by the system
Figure 2. PlayWrite Interface. (a) Grab to Talk: picking up a character triggers user’s voice input. (b) Drag to Move: repositioning the character’s transform handle for moving them in space with character animations. (c) Attach a Prop: objects can be linked to characters to contextualize actions. (d) AI Respond: the system generates dialogue and narrative continuation in response to user actions. (e) Marble Assembly: users arrange story marbles on a timeline, each representing an Intent Frame, to sequence and refine the story. Dialogue and character locations are replayed when a marble is selected, supporting reflection and revisiting. Different views of the PlayWrite interface layout. The main 3D playground shows characters and props. A dialogue window appears above characters when they speak. Below, a horizontal timeline displays circular “story marbles” that can be selected and rearranged.
Figure 3. System pipeline of PlayWrite . (1) Multi-agent observation collects user inputs (speech, grabbing, moving) and interprets them through Environmental, Social, and Narrator agents. (2) The Intent Frame Agent fuses these into higher-level features through temporal filtering, contextual prioritization, and cross-agent fusion. (3) Intent frames are assembled into marbles, which can be replayed, reordered, or exported. (4) Story generation produces a coherent story summary and screenplay from the assembled marbles. Flowchart-style diagram showing PlayWrite’s pipeline. On the left are user inputs (grabbing, moving, talking). In the center are three types of agents (Environmental, Social, Narrator). Arrows lead to Intent Frame generation and then to Story Assembly and Story Generation. Small icons of characters and objects appear alongside each stage.
Figure 4. Examples of semantically meaningful features captured during play and encoded into Intent Frames. (a) Zone of Interest: a character placed within a predefined hiding zone, and therefore, his action and speech will not be noticed by other characters. (b) Movement Trail: the movement of a character is tracked to capture directionality and pacing. (c) Character–Prop Interaction: a gun is attached to signal a threat. Other characters will react to seeing the gun in hand. (d) Character–Character Interaction: relative positioning and engagement between characters are recorded as two characters relaxing together in the pool. (e) Proximity: spatial closeness between the character and a magic carpet indicates plot progression. Together, these features provide structured cues that the system abstracts into Intent Frames for narrative interpretation. Exemplary interactions: sequence of frames showing a user arranging characters, triggering speech, assembling Story Marbles, and exporting a screenplay.
Figure 5. Recorded example workflow of Robin Hood extracted from a user’s play (P6, open-ended task). Green-labeled boxes show user speeches, and black-labeled boxes show AI responses demonstrating two conversation segments: (first row) Mary pleads with Robin Hood for help. The user, controlling Robin, confronts Lord Pemberton. (second row) The story develops as the user switches characters and overrides AI responses to introduce a plot twist. Complete dialogue and synopsis can be found in Appendix E and F. Exemplary interactions: sequence of frames showing a user arranging characters, triggering speech, assembling Story Marbles, and exporting a screenplay.
| ID | Writing Background | Primary Writing Format |
|---|---|---|
| P1 | Songwriting, interactive media storytelling, fantasy, science fiction, screenwriting, game narrative, poetry | Scripts/screenplays |
| P2 | Poetry, journals, literary fiction | Short stories |
| P3 | Fiction writing, poetry, literary fiction | Short stories |
| P4 | Prose fiction, fantasy, science fiction, poetry | Short stories |
| P5 | Science fiction writing, screenwriting, game narrative | Short stories |
| P6 | Dungeon master, improv comedy, game narrative, interactive fiction | Improv comedy and D&D narratives |
Table 1. Participant backgrounds, mapped to the anonymized IDs (P1–P13) used when reporting quotes and findings. Table listing each participant with demographics such as writing background and primary storytelling medium.
Figure 6. Three distinct scenes used in the user study. (a) Tutorial Scene: A simple environment depicting an interview between two characters with a desk zone and stage zone, including an Elvis impersonator, a hiring manager, and basic props. This scene was used to introduce participants to the system’s core interactions (grabbing, moving, speaking, attaching props). (b) Aladdin Scene: A desert-themed environment with pond and hide zones, featuring Aladdin, Genie, and Jafar alongside interactive props. Participants were encouraged to improvise and stage interactions using familiar characters. They were presented with an end goal to reach. (c) Robin Hood Scene: A castle environment with an open space zone, featuring Robin Hood, Mary, and Lord Pemberton, along with thematic props. Participants were instructed to use the characters and the props for an open-ended narrative exploration. Together, these three scenes provided varying levels of narrative structure, scaffolding participants from practice to open-ended improvisation. Three different scenes shown side-by-side. Each has unique characters, props, and a distinct environment theme. The images represent the pre-authored starting scenes used in the study.
| Factor | Mean | SD |
|---|---|---|
| Expressiveness | 6.35 | 0.77 |
| Enjoyment | 6.15 | 0.80 |
| Immersion | 5.85 | 1.41 |
| Results Worth Effort | 5.85 | 1.07 |
| Exploration | 5.38 | 1.43 |
| Control | 3.92 | 1.93 |
Table 2. Creativity Support Index (CSI) Results (N=13) Table showing Creative Support Index (CSI) subscale scores (e.g., Enjoyment, Exploration, Control).
Figure 7. Results of post-study questionnaires for: ( a ) Overall Experience, ( b ) Co-creative Partnership, and ( c ) Perceived Playfulness. Results of post-task questionnaires for: ( d ) Closed-Ended Task and ( e ) Open-Ended Task a) Bar chart showing participants’ responses to co-creativity and playfulness questions. The overall results are positive, with lower agreement in items related to precise control of the AI. b)Comparison bar chart across two storytelling modes: goal-oriented vs. open-ended. Each mode has multiple response bars showing different aspects of user experience; differences are visible in perceived exploration and challenge.
Figure 8. Action-level interaction analysis across participants. ( a ) Raw action beats for the Aladdin scenario, showing participant timelines with color-coded action events. ( b ) Stacked distribution of action types across participants in the Aladdin scenario. ( c ) Raw action beats for the Robin Hood scenario. ( d ) Stacked distribution of action types across participants in the Robin Hood scenario. Together, these visualizations highlight how participants combined speech and physical actions over time, and how relative reliance on each interaction type varied across individuals. The periods of user inactivity have been removed from the raw action beat visualizations (a, c) to focus on active interaction sequences. Stacked bar chart showing interaction types used by participants (e.g., grabbing, speech, moving). Colors differentiate interaction categories, and bars vary in length, showing differences in behavior and preference across participants.