Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to input prompt variations. This paper presents \unicodex00ABPromptResponse\unicodex00BB, a controlled study examining how formatting and LLM-based tuning of coding task prompts affect the resulting code's performance, efficiency, and stability. Using five semantically identical yet syntactically distinct variants of the HumanEval dataset\unicodex2014baseline, JSON, Markdown, YAML, and an LLM-tuned version\unicodex2014we had GPT-4o solve its coding problems over 8200\unicodex00A0executions. Our results show that consistent formatting\unicodex2014especially JSON\unicodex2014improves generation efficiency and syntactic stability, with minor gains in task performance. Conversely, the LLM-tuned prompts resulted in significantly degraded task performance without significant improvements in any other dimension. These findings suggest that low-effort reformatting alone can yield measurable improvements, while tuning must account for model alignment. We conclude our work with providing a set of practical recommendations informed by our results as well as releasing our dataset variants and evaluation pipeline for future work.
Figures & tables
Figure 1
Figure 1. A prompt designed to highlight some of the inconsistencies present in the HumanEval dataset, before and after possible reformatting. Which one will lead to better results?
Figure 3Figure 4Figure 5Figure 6Figure 7Figure 8
#
Role
Content
0
system
You are an expert prompt rewriter. Your task is to improve the clarity and helpfulness of docstrings for code generation. Only rewrite the docstring (the text inside triple double quotes: """ … """). Do not modify the function signature or write any implementation code.
1
user
<prompt>
Table 1. The messages each MistralAI API request for creating the ‹tuned› dataset consisted of.
#
Role
Content
0
system
You are a Python programming expert.
1
user
Please solve the following problem. Output only the function with the signature as specified in the prompt and all necessary imports. Do not include additional texts or comments. <prompt>
Table 2. The messages each API request consisted of, with a prefix for orientating the LLM before any given <prompt> .
Figure 2. The PassRate across all five levels of dataset , with each dot representing the aggregate result of a prompt’s 10 executions.
Figure 3. The GenDuration across all five levels of dataset , aggregated over the 10 executions per prompt.
Figure 4. The EvalDuration across all five levels of dataset , aggregated over the 10 executions per prompt.
Figure 5. The PassDuration across all five levels of dataset , aggregated over the up to 10 passing executions per prompt.
Figure 6. The ResponseLen across all five levels of dataset , aggregated over the 10 executions per prompt.
Figure 7. The averages of the pairwise Rouge-L scores across all five levels of dataset , with each dot representing a prompt’s 10 executions.
Large language models (LLMs) are widely used for open-ended tasks, but underspecified prompts can lead to low-quality answers and additional interaction. This paper studies whether structured prompt design improves response quality while reducing user effort. We compare three prompt conditions: a raw prompt, a checklist-improved prompt, and a clarifying-question prompt. We evaluate these conditions across four task types--summarization, planning, explanation, and coding--using three LLM systems: ChatGPT, Claude, and Grok. Each output is scored with a unified rubric covering task completion, correctness, compliance, and clarity. Checklist-improved prompts achieved the highest mean rubric score, 7.50 out of 8, compared with 5.67 for raw prompts and 6.67 for clarifying-question prompts. Checklist prompts also produced the best quality-effort tradeoff, using fewer average tokens than both raw and clarifying prompts. These results suggest that a simple prompt checklist can improve LLM responses while reducing unnecessary interaction.
Saurav Ghosh, Gabriella Polach, Abdou Sow
Washington University in St. Louis Saint Louis, Missouri, USA
Large Language Models (LLMs) are increasingly integrated into software engineering workflows, helping developers write, debug, test, and maintain code. While prompt wording and structure are known to influence model performance, the impact of psychologically inspired prompt framings remains unexplored. This study investigates whether different psychology-based communication strategies that humans use to persuade or motivate others can lead to more effective prompt framing, which may, in turn, affect LLM behaviour in coding tasks. Drawing on Yukl & Falbe's well-known taxonomy, we operationalized eight influence tactics (like rational persuasion, ingratiation, and exchange) into reproducible prompt templates. These prompt templates were evaluated across five leading open-weight LLMs using two widely adopted benchmarks: LiveCodeBench and SWE-bench Verified. We assessed the resulting code output on four key software quality dimensions: functional correctness, quality, maintainability, and security. Our results show that certain influence-induced prompt framings, particularly those emphasizing urgency, were associated with reduced correctness and security. This work presents the first large-scale empirical study of influence-induced prompt framing in software engineering tasks, offering insights into how linguistic cues may shape LLM outputs. We conclude with practical insights for designing transparent and interpretable human-AI interactions in code generation.
Alex Deaconu, Anubhav Gupta, Manaal Basha +2
Alex Deaconu and Anubhav Gupta shared first authorship.
The behavior of contemporary generative Large Language Models (LLMs) is directly shaped by prompts, unstructured texts that describe the desired output and model behavior. In this paper we argue that prompts are linguistic objects that merit investigation in their own right. To this end, we collect 57.5K unique samples of prompts from GitHub. Specifically, we focus on transactional prompts: reproducible natural language instructions that are integrated into software. To enable the empirical, quantitative study of prompts, we introduce a structured ontology, capturing the properties of prompts as well as their formal and semantic components. Based on this ontology, we transform prompts from unstructured raw texts into richly structured linguistic objects. Analysis of these structured data reveals significant diversity of usage patterns across languages, domains, tasks, and modalities, in a typical Zipf-like distribution where some clearly prevail and others, more diverse, appear in the long tail. To validate the reliability of the ontology-based annotation of the prompts, we perform a comprehensive error analysis across all fields, providing a detailed assessment of annotation quality. We release the dataset together with a browsing and exploration interface (https://github.com/OnlpLab/transactionalPromptsCollection ).
Victoria Basmov, Yoav Goldberg, Reut Tsarfaty
Bar-Ilan University · Allen Institute for Artificial Intelligence