Understanding Developer Information Needs for Software with Embedded Natural Language Prompts
Organizations: Carnegie Mellon University, Pittsburgh, Pennsylvania, USA · JetBrains Research, Belgrade, Serbia
Abstract
Developers use foundation models (FMs) to create novel forms of intelligent software by writing natural language prompts embedded within a software's implementation---referred to as embedded prompts---that are also passed as a string within an FM's API. Developing software with embedded prompts requires developers to engage in iterative embedded prompt changes to obtain the desired behavior. While the information needs of traditional software have been investigated, the information needs of modern developers for embedded prompt changes are underexplored, despite these needs shaping how developers plan and justify embedded prompt changes. In this work, we perform a mixed-method exploratory study on the information needs of developers making changes to embedded prompts. We create, refine, validate, and apply a list of 50 information needs that developers have while making a change to an embedded prompt by interviewing 16 developers, observing 8 developers make prompt changes, surveying 50 developers, and evaluating 72 research and commercial tools. We find that embedded prompts are not well-supported by existing tools: 16 of the 50 information needs, including four of the most important ones, are unaddressed. We conclude with important design opportunities that should be addressed by future developer tools for embedded prompts.
Figures & tables
| ID | Context | Exp. | # Prompts | Structure | Model | App Domain | Task |
|---|---|---|---|---|---|---|---|
| IP1 | Academia | 16 yrs | 30 | Single prompt | \chip OS \chip LM | Security | Classification |
| IP2 | Personal project | 10 yrs | 40 | Single prompt | \chip CS \chip LM | Productivity | Text summarization (chatbot) |
| IP3 | Academia | 8 yrs | 30 | Single prompt | \chip CS \chip LM | Research analysis | Classification |
| IP4 | Big tech (R&D) | 8 yrs | 10 | Single prompt | \chip OS \chip LM | Software testing | Code generation |
| IP5 | Academia | 14 yrs | 35 | Single prompt | \chip CS \chip LM | Education | Generation |
| IP6 | Academia | 10 yrs | 3 | Multi-prompt | \chip CS \chip LM | Software testing | Code generation |
| ID | Prog. Exp. | Prof. Exp. | # Prompts | Structure | Model | App Domain | Task |
|---|---|---|---|---|---|---|---|
| OP1 | 10 yrs | 5 yrs | 4 | Agent | \chip CS \chip VLM | Security | Classification |
| OP2 | 11 yrs | 7 yrs | 100+ | Agent | \chip CS \chip LM | Software eng. | Code refactoring |
| OP3 | 6 yrs | 3 yrs | 1 | Agent | \chip CS \chip LM | Software eng. | Code generation |
| OP4 | 25 yrs | 0 yrs | 25 | Multi-prompt | \chip CS \chip LM | FM reasoning | Generation |
| OP5 | 5 yrs | 1 yr | 4 | Single prompt | \chip CS \chip LM | Astrology | Generation |
| OP6 | 5 yrs | 4–5 yrs | 100+ | Single prompt | \chip CS \chip LM | Education | Q&A (chatbot) |
| Category | Tools |
|---|---|
| Traditional office tools | Online workspaces (e.g., Notion); Spreadsheets (e.g., Google Sheets); Text processors (e.g., Google Docs, Notepad) |
| Coding tools | Code editors; Computational notebooks (e.g., Jupyter); In-IDE programming assistants (e.g., GitHub Copilot); Version control systems (e.g., Git, GitHub) |
| Prompting GUIs | Chat interfaces (e.g., ChatGPT); Prompt playgrounds (e.g., OpenAI Chat Playground) Anthropic Developer Platform : Playground; Skills; Agents OpenAI Developer Platform : Chat; Decisions; Agents; Audio; Images |
| Prompt repositories | LangChain Hub ( lan, 2025 ) ; Prompt Flow ( mic, 2026 ) |
| Popular open-source prompting tools | Agenta ( git, 2026a ) ; Helicone ( git, 2026b ) ; Humanloop ( git, 2026c ) ; Langfuse ( git, 2026e ) ; LangSmith ( git, 2026d ) ; Langtrace ( git, 2026j ) ; MLflow ( git, 2026h ) ; Prompt Flow ( git, 2026g ) ; PromptFoo ( git, 2026i ) ; PromptLayer ( git, 2026f ) |
| Research tools | AI Chains ( Wu et al., 2022b ) ; AP-Adapter ( Fu et al., 2024 ) ; AutoStruct, AutoStruct+, SelectStruct, TagStruct ( Fiannaca et al., 2023 ) ; BotDesigner ( Zamfirescu-Pereira et al., 2023a ; Zamfirescu-Pereira et al., 2023b ) ; CoPrompt ( Feng et al., 2024 ) ; CoPrompter ( Joshi et al., 2025 ) ; DPO-Diff ( Wang et al., 2024c ) ; G-fair-Prompting, T-fair-Prompting ( Ma et al., 2023 ) ; Graphologue ( Jiang et al., 2023 ) ; ChainForge ( Arawjo et al., 2024 ; Arawjo et al., 2023 ) ; ConstraintMaker ( Liu et al., 2024 ) ; EASE ( Wu et al., 2024 ) ; EvalLM ( Kim et al., 2024 ) ; MoP ( Wang et al., 2024a ) ; Partial2Global ( Xu et al., 2024b ) ; PE2 ( Ye et al., 2024 ) ; ProMaC ( Hu et al., 2024a ) ; PromptAid ( Mishra et al., 2025 ) ; Promptbreeder ( Fernando et al., 2024 ) ; PromptCanvas ( Amin et al., 2025 ) ; PromptChainer ( Wu et al., 2022a ) ; PromptCharm ( Wang et al., 2024b ) ; PromptIDE ( Strobelt et al., 2022 ) ; PromptingSheet ( He et al., 2025 ) ; PromptHive ( Reza et al., 2025 ) ; Promptify ( Brade et al., 2023 ) ; Promptist ( Hao et al., 2023 ) ; PromptMaker ( Jiang et al., 2022 ) ; RePrompt ( Wang et al., 2023 ) ; Reprompting ( Xu et al., 2024a ) ; Responsible Prompting Recommendation ( Santana et al., 2025 ) ; RLPG ( Shrivastava et al., 2023 ) ; Smart Prompt Advisor ( Phokela et al., 2023 ) ; Synthetic Prompting ( Shao et al., 2023 ) ; TRIPLE ( Shi et al., 2024 ) ; UP-DP ( Li et al., 2023 ) ; WorldSmith ( Dang et al., 2023 ) ; ZOPO ( Hu et al., 2024b ) ; Gero et al. (2022) ; Gero et al. (2024) ; White et al. (2023) ; White et al. (2024) |
| Task (T#) | Question (Q#) | Freq. | Help. | Addr. |
| What is this prompt? | ||||
| T1. Understanding a version’s content | Q1. What is the high-level structure of the prompt content? | 63% | 53% | |
| Q2. What is the specific text (e.g., characters) in the prompt content? | 56% | 40% | ||
| Q3. What other parts of the prompt are related to the part I’m interested in? | 92% | 80% | ||
| T2. Understanding a version’s behavior | Q4. What output does this prompt generate for this example? | 87% | 68% | |
| Q5. How is this prompt behaving or performing overall? | 85% | 77% | ||
| Distribution | ||||||
| Task (T#) | Frequency | Helpfulness | ||||
| What is this prompt? | ||||||
| T1. Understanding a version’s content | 49% | 21% | 39% | 6% | ||
| T2. Understanding a version’s behavior | 63% | 7% | 59% | 5% | ||
| T3. Identifying examples with interesting behavior | 63% | 15% | 68% | 5% | ||
| T4. Hypothesizing reasons for prompt behavior | 68% | 10% | 61% | 12% | ||
AI-Assisted Software Development
AI-assisted software development uses language models and coding agents to help people design, write, test, review, and maintain software.
See all 115 papers