cs.HCSep 18, 2026
SaveGricea: An Open Science Platform for Conversational AI Research
Organizations: Johns Hopkins University Baltimore, MD, USA
Abstract
We need studies on conversational AI (CAI) at scale to understand human behavior and shape CAI design. However, fragmented reporting of systems and study configurations hinders replication, extension, and knowledge accumulation. We present Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Informed by a formative analysis of prior CAI research, Gricea couples study procedures, participant-facing systems, and conversational task behavior in. In a replication study using Gricea, we replicated configurations 93% of eligible CUI 2026 papers; while also flagging missing information in 96% of papers that hinder faithful replication --- further motivating Gricea's need. In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies addressing various open-ended research questions. Together, these findings demonstrate Gricea's support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science.
Figures & tables
Figure 1. A shared research foundation for conversational AI. A: Gricea represents a study through connected Study Flow and Task Flow definitions, preserving both configured choices and unchanged defaults in an inspectable, executable artifact. B: Other researchers can inspect and extend the artifact by changing its grounding context while retaining the remaining study design; The updated study can be deployed and shared. C: The research community and AI agents can build on shared executable artifacts across successive studies, accumulating new findings into collective knowledge Three panels show the lifecycle of a shared study artifact. A research configures a between-subject study using connected Study Flow and Task Flow graphs. Another researcher inspects that artifact, adds their configuration edit to the study while retaining other settings, and publishes a new version. The research community can through a shared executable representation builds collective knowledge.
| Dimension | Examples of study configuration | Implications for study infrastructure |
|---|---|---|
| Interface Condition | Layout and workspace organization; citation and provenance display; highlighting; progress guides; verification prompts; structural overlays; editability of shared artifacts; time-synchronized guidance and conversation navigation ( Chen et al., 2025 ; Cheng et al., 2024 ; Jeong et al., 2025 ; Chi et al., 2025 ; Nguyen et al., 2025 ) . | Represent what participants can see, inspect, and change. Preserve presentation and available actions as part of the study condition, including how evidence, guidance, and task progress are displayed. |
| Agent Condition | System instructions; persona or role framing; stance; response style; initiative; clarification behavior; trust-adaptive interventions; tool-use policy; response delays and waiting cues; coordination of model components ( Jakesch et al., 2023 ; Dhillon et al., 2024 ; Li et al., 2025 ; Liu et al., 2025 ; Maslych et al., 2025 ; Liang and Luo, 2025 ) . | Make agent behavior, permitted actions, timing, and coordination explicit. Distinguish the roles participants encounter from the model components that implement those roles. |
| Context & Grounding | Retrieved documents; retrieval policy; context filtering; citation constraints; external evidence injection; opposing or balanced context; task context; interaction history; persistent memory and its inspection or editing ( Lewis et al., 2020 ; Sharma et al., 2024 ; Sharma et al., 2025 ; Chi et al., 2025 ; Yan et al., 2025 ) . | Specify what information the agent receives, what participants can inspect or change, and how context persists across turns or sessions. Distinguish information available to the agent from history displayed to participants and records retained for research. |
| Task & Modality | Search, writing, tutoring, coding, annotation, voice, multimodal interaction, direct artifact manipulation, and conversational control of haptic feedback ( Qian et al., 2025 ; Yao et al., 2026 ; Flores-Saviaga et al., 2025 ; Caetano et al., 2025 ; Mishra and Schneider, 2025 ) . | Connect different participant activities and modality-specific inputs and outputs to a shared study procedure. Support task-specific actions and outcome collection without rebuilding the surrounding study infrastructure. |
| Domain & Audience | Topic domain; expertise; prior beliefs and experience; participant population; targeted community; high-stakes versus low-stakes settings; language and dialect; accessibility and care needs ( Pang et al., 2025 ; Rathod M.S., 2024 ; Ge et al., 2024 ; Flores-Saviaga et al., 2025 ; Elwahsh et al., 2025 ; Czech et al., 2025 ) . | Document the task setting, study materials, and relevant participant characteristics separately from assigned conditions. Preserve the context needed to interpret findings and adapt a study for another population or setting. |
| Study Procedure | Pre-task elicitation; condition assignment; branching; within- or between-subject structure; counterbalancing; post-task measures; longitudinal follow-up; scheduled check-ins and cross-session continuity ( Jakesch et al., 2023 ; Qian et al., 2025 ; Cai et al., 2023 ; Abbas et al., 2025 ; Cox et al., 2025 ; Penney et al., 2025 ) . | Make assignment, task order, branching, measurement timing, and cross-session continuity explicit and authorable. Preserve the procedure as executable logic that can be inspected, reused, and extended. |
Table 1. Recurring dimensions of conversational AI study configuration, synthesized from the formative analysis. Examples include experimental manipulations, fixed settings, and participant characteristics. The infrastructure implications identify what must remain explicit to design, inspect, reproduce, and extend a study.
Figure 2. The Gricea authoring interface, illustrating how researchers can assemble multi-stage study procedures (left) and independently configure agent pipelines and interface scaffolds for specific conversational tasks (right). The Gricea authoring interface, illustrating how researchers can assemble multi-stage study procedures (left) and independently configure agent pipelines and interface scaffolds for specific conversational tasks (right).
Figure 3. Participant-facing interfaces configured in Gricea : conversational AI with integrated surveys (top left), real-time voice interaction alongside a driving task (top right), conversational coding with an editable code workspace (bottom left), and a participant interacting with two conversational agents (bottom right). Four screenshots show participant-facing interfaces. The top-left interface combines survey questions, a text conversation, and a response-rating item. The top-right interface places a driving simulation beside voice interaction controls. The bottom-left interface places a code editor with proposed changes beside an AI assistant. The bottom-right interface displays separate panels for Agent A and Agent B, with indicators for participant and agent turns.
| Participant | Open-ended research question |
|---|---|
| P01 | Under what conditions and tasks do users rely on different agentic tool calls, such as web search? |
| P02 | How does serendipity differ across web search, generative-AI search, and retrieval-augmented generation? |
| P03 | How do proactive versus reactive coding agents shape participants’ coding practices? |
| P04 | How does AI-assisted creative writing compare to human-assisted creative writing? |
| P05 | If people cannot distinguish AI-written text from human-written text, where does the perceived “AI penalty” in evaluation come from? |
| P06 | How do different personas of AI chatbots affect engineers in mechanical design tasks? |
Table 2. Open-ended research questions participants implemented as valid, runnable Gricea studies. These studies varied across procedures, interfaces, models, surveys, and outcomes, illustrating the range of research designs expressible through the shared study representation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Role | Conducted studies? | Study count | Primary research interests / intended study |
|---|---|---|---|---|
| P01 | Researcher | Yes | 5–10 | Capability boundary of conversational AI; what tasks can and cannot be automated; human–human / human–agent boundaries. |
| P02 | Master Student | No | N/A | Serendipity in search; comparing web search, GenAI search, and RAG search; detailed search-session logging. |
| P03 | Master Student | No | N.A. | Legal AI and hallucinated precedent; coding agents; mental-health advice and human influence. |
| P04 | PhD | Yes | 1–5 | Social consequences of conversational agents; tutor-like interaction; transfer from human–AI to human–human interaction. |
| P05 | PhD | Yes | 1–5 | AI penalty: how judgments change when people learn a text was AI-generated versus human-generated. |
| P06 | PhD | Yes | 1–5 | Utility of AI systems in design / mechanical collaboration; AI personas in design tasks. |
Table 3. Completed participant records used in the analysis ( usable completed records in the attached JSON export). Participants spanned multiple roles and brought diverse conversational-AI study ideas to the authoring task.
| Reference | Paper |
|---|---|
| ( Bieberstein et al., 2026 ) | Do Prompt-Level Empathy Instructions Influence User Experience? Evidence From A Controlled Chatbot Study |
| ( Liu et al., 2026c ) | AI Echoing in the Backstage: Private AI Consulting May Strengthens Confidence and Limits Depolarization |
| ( Abdrabou et al., 2026 ) | Beyond Benchmarks: A User-Centric Framework for Evaluating Large Language Models |
| ( Spillner et al., 2026 ) | Linguistic Uncertainty Markers for Trust Calibration in AI-Assisted Decision-Making |
| ( Seymour et al., 2026 ) | Beliefs and Misconceptions around Integrated Conversational AI |
| ( Rekkas et al., 2026 ) | Design and Evaluation of ChatBlend: A Framework and Card Deck for the Human-Centered Design of Mental Health Chatbots in Blended Care |
Table 4. The complete corpus of 29 eligible CUI 2026 papers that we attempted to recreate.
More in this topic
Scientific Reproducibility
Scientific reproducibility studies whether research methods, data, and software can be inspected and rerun to support published findings.
See all 108 papers