Organizations: SolJAMM Research and ATLAS Institute University of Colorado, Boulder Boulder, USA · University of Colorado, Boulder Boulder, USA · ATLAS Institute University of Colorado, Boulder Boulder, USA · Dep. of Information Engineering and Computer Science University of Trento Trento, Italy
Technology-mediated audience participation (TMAP) enables audiences to influence live performances, yet balancing interaction with meaningful agency remains challenging. While AI-generated music and MR enable new forms of audio and visual audience participation, little is known about how these participation modalities influence audience experiences during live musical performance. In particular, it remains unclear how they shape perceived agency, collaboration, and engagement, and how they complement one another within TMAP systems. We present CrowdForge, an AI- and MR-mediated paradigm where audiences collectively control generative music and virtual environments during a live performance featuring a guitarist and DJ. Through a mixedmethods concert study evaluating three interaction conditions (audio, visual, and combined audio-visual), we demonstrate that observability is the primary driver of audience agency. While visual interaction provided easily observable feedback, audio parameters offered macro-structural control over the music. Audiences tended to use the control enabled by AI audio generation to test musical boundaries, introducing a productive adversarial tension between the crowd and the musicians. This dynamic pushed creative limits, deepened engagement, and generated moments of musical coherence when performers successfully resolved audience-induced challenges. We argue future TMAP systems should prioritize human-centered, highly observable interactions to build agency, while combining the expressiveness of music with the clarity of visual feedback.
Figures & tables
Fig. 1: Overview of the audience participation workflow in CrowdForge. (A) Audience selections are aggregated into a collective configuration spanning musical character, tempo, instrumentation, dancer movement, virtual performers, and stage environment. (B) The selected parameters are incorporated into a live musical performance involving human musicians and projected audiovisual content. (C) The visual selections simultaneously configure a mixed-reality performance environment populated by virtual musicians and dancers. Together, these audio and visual modalities allow audience input to become a perceivable component of the resulting performance.
Fig. 2: System diagram and signal flow illustrating the control audience members had over elements of the performance (from left to right). Audience members visit a web page on their mobile devices and vote on parameters they want to see reflected in the performance. The inputs are then aggregated and sent to both a generative transformer model (played through Ableton Live 12 software) and a mixed reality engine (Unity 3D) for rendering both audio and visuals.
Fig. 3: A) Mobile user interface for audience member during the audio parameter selection condition. B) Sample projected aggregator screen displayed to audience after selections were made.
Condition (No. of Parameters)
Available Options
Audio (14) - Genre / Feel - Tempo - Mode - Human Guitar Tone - Instruments - Piano FX - Guitar FX - Bass FX - Strings FX - Winds FX
Visual (14) - Environment - Character Skins for 9 characters - Dance Moves for 2 dancers
Environment: haunted forest, mars scape, laser grid Character Skins: vampire, werewolf, moose orc, goblin elf, flesh eater, dark noire, lizard species, webbed and armored, x-bot, y-bot Dance Moves: hiphop, breakdance, latin, party
Both (28) - All Audio Parameters - All Visual Parameters
Description: The “both” mode combines all available parameters from the audio and visual conditions.
TABLE I: Parameters and Available Options per App Condition
Fig. 4: Indicates dominance, valence, and arousal scores on the self-assessment manikin, comparing Audio, Visual, and Both conditions. Each measure is rated 1-9. Significance figures are demonstrated with respect to the neutral baseline of 5 (shown in red).
Fig. 5: Pairwise testing indicates a significant difference between the Visual and Both conditions ( p=0.015 ). Red line indicates neutral score.
Fig. 6: Indicates perceived influence and agency on the performance when selecting specific audio elements. Both measures use a rating scale of 1-7. Significance figures and effect sizes (r) are demonstrated with respect to the neutral baseline of 4 (shown in red).
Music and dance are social practices of expression and connection, yet most HCI work in human-AI co-creation centers the solo performer. As generative music matures, we ask not only what AI can compose but what social encounters it can organize around sound. We present Encypher, a collaborative generative music system that translates collective movement qualities into text prompts conditioning real-time music generation for dance cyphers. Through five weeks of co-design with local dancers, a user study with unacquainted participants, a public museum event, and a live performance, we found that users developed shared agency, perceiving the music as a response to the room's energy. While newcomers felt uncertain, the system fostered social presence by prompting them to look to each other for cues. By treating sociality as a design concern rather than a downstream effect, we offer a framework and design implications for AI systems for collaborative, embodied expression.
Zhixing Chen, Cheng-Zhi Anna Huang
Massachusetts Institute of Technology Cambridge, USA
Melodic material in Hindustani music is presented in relation to a tonic, usually sustained by the tanpura, a four-stringed drone instrument. Rooted in Hindustani music, 'The Moving Drone' sets the traditionally static drone into motion that, throughout the performance, gains increasing agency transitioning from reactive to more proactive roles. The work employs four independent loopers in Max/MSP to function as 'virtual' drones. They are populated cyclically in real-time as the vocalist improvises, creating an organic and evolving feedback loop between the voice and the virtual drone. This relationship further evolves melodically by pitch shifting the loops, which introduces a dimension of sudden, explicit movement. Then it changes timbrally, via the integration of GaMaDHaNi, a singer conditioned pitch-to-voice generative AI model to resynthesize looped audio. While current music AI approaches prioritize high-fidelity and realism of generated content which has sparked anxiety over job replacement for the music community, this work intentionally utilizes low-fidelity generative outputs, further necessitating human interpretation and situational context in order to be complete. 'The Moving Drone' positions technology and generative AI within established socio-cultural musical practices, proposing a virtual drone as an active, responsive, and co-creative musical agent.
Nithya Shikarpur, Victor Arul, Anna Huang
Massachusettes Institute of Technology, USA · Harvard University, USA
A concert is a collective experience, but recorded performance videos are typically watched alone, stripping away the shared audience presence that makes concerts feel eventful. We investigate whether persona-based LLM audience agents can recreate aspects of this collective experience by generating real-time fan chat alongside a K-pop performance video. We present a multi-agent system in which ten LLM agents react through live-chat messages, comparing a persona-conditioned audience (each agent assigned a distinct fan identity, bias, and chat style) with a no-persona baseline. In a within-subjects pilot with K-pop fans (N=11), persona conditioning substantially improved model-level chat quality and perceived naturalness, but did not translate into differences in social connectedness, engagement, or affective response. Interviews suggest that online K-pop concert chat may operate as collective monologue rather than interpersonal dialogue, and that meaningful participation depends on shared identification with the specific artist and fandom. Persona conditioning can make LLM audiences appear more natural, but culturally meaningful collective experience may require deeper alignment between persona, crowd behavior, fandom identity, and user expectations.
Kirak Kim, Hyojin Kim, Yejin Son +2
Graduate School of Culture Technology, KAIST, Daejeon, South Korea · Department of Artificial Intelligence, Yonsei University, Seoul, South Korea