Prompt Sensitivity

Ask about this topic

Overview

Prompt sensitivity studies how models respond to changes such as paraphrases, word order, option labels, examples, schemas, and contextual framing, including when the intended meaning stays the same. Researchers measure output flips, score variance, consistency, and ranking changes across controlled prompt variants, and use prompt aggregation, robustness training, and internal-state analyses to explain or reduce these effects. The issue spans language, vision-language, clinical, safety, and benchmark settings, where prompt choices can affect both correctness and behavior. A central tension is that greater consistency does not guarantee correctness or safety, and methods that help in one model or task may fail in another.

Latest papers

CardsList

115 papers18 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

  1. Testing the Limits of Truth Directions in LLMs

    Apr 4, 2026Angelos Poulis, Mark Crovella, Evimaria TerziLLM InterpretabilityLinear Probing

    Department of Computer Science, Boston University · Faculty of Computing & Data Sciences, Boston University

  2. LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

    Mar 31, 2026Filip J. Kucia, Anirban Chakraborty, Anna WróblewskaAutomated Essay ScoringLanguage Model Bias Evaluation

    Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK

  3. Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

    Mar 30, 2026Doan Nam Long Vu, Simone BalloccuMedical VLMsPrompt Sensitivity

    Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt

  4. ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation

    Mar 27, 2026Xianpeng, Sun, Haonan Sun +5Software Engineering BenchmarksTemporal Data Leakage

    Simon

  5. A Regression Framework for Understanding Prompt Component Impact on LLM Performance

    Mar 27, 2026Andrew Lauziere, Jonathan Daugherty, Taisa KushnerLLM PromptingLLM Interpretability

    Galois, Inc.

  6. When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

    Mar 26, 2026Binesh Sadanandan, Vahid BehzadanPrompt SensitivityLLM Prompting

    SAIL Lab, University of New Haven, West Haven, CT 06516, USA

  7. How Utilitarian Are OpenAI's Models Really? Replicating and Reinterpreting Pfeffer, Krügel, and Uhl (2025)

    Mar 24, 2026Johannes HimmelreichPrompt SensitivityMoral Reasoning in Language Models

  8. Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models

    Mar 22, 2026Binesh Sadanandan, Vahid BehzadanVLM EvaluationMedical VLMs

    SAIL Lab, University of New Haven, West Haven, CT, USA

  9. Is Evaluation Awareness Just Format Sensitivity? Limitations of Probe-Based Evidence under Controlled Prompt Structure

    Mar 19, 2026Viliana DevbunovaEvaluation Awareness in Language ModelsLanguage Model Probing

    Yandex Belgrade, Serbia

  10. Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectors

    Mar 19, 2026Momose Oyama, Yusuke Takase, Hidetoshi ShimodairaPrompt Sensitivity

    Kyoto University · RIKEN

  11. When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making

    Mar 16, 2026Nazia RiasatLLM EvaluationLLM Reliability

    North Dakota State University

  12. Evaluation format, not only model capability, shapes measured triage failure in consumer health AI

    Mar 12, 2026David Fraile Navarro, Jialei Sheng, Farah Magrabi +1Language Model Safety EvaluationLLM Safety Benchmarks

    Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia · NSW Health, Sydney, NSW, Australia

  13. Disentangling Prompt Dependence to Evaluate Segmentation Reliability in Gynecological MRI

    Mar 9, 2026Elodie Germani, Krystel Nyangoh-Timoh, Pierre Jannin +1Prompt SensitivityPromptable Image Segmentation

    UR, LTSI · Laboratoire Traitement du Signal et de l’Image (LTSI, INSERM UMR 1099), Université de Rennes, Rennes, France · LTSI +1

  14. The Fragility Of Moral Judgment In Large Language Models

    Mar 5, 2026Tom van Nuenen, Pratik S. SachdevaMoral Reasoning in Language ModelsPrompt Sensitivity

    D-Lab, University of California, Berkeley, USA

  15. When Do Language Models Endorse Limitations on Human Rights Principles?

    Mar 4, 2026Keenan Samway, Nicole Miu Takagi, Rada Mihalcea +4LLM AlignmentLanguage Model Bias Evaluation

    Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · University of Michigan +2

  16. Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare

    Mar 1, 2026Shravani HariprasadClinical Language Model EvaluationPrompt Sensitivity

    Independent Researcher

  17. Brittlebench: Quantifying LLM robustness via prompt sensitivity

    Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt SensitivityLLM Evaluation

    FAIR at Meta · EPFL · Northeastern University

  18. Scaling In, Not Up? Testing Thick Citation Context Analysis with GPT-5 and Fragile Prompts

    Feb 25, 2026Arno SimonsScientific CommunicationPrompt Sensitivity

    Technische Universität Berlin

  19. What Makes a Good Query? Measuring the Impact of Human-Confusing Linguistic Features on LLM Performance

    Feb 23, 2026William Watson, Nicole Cho, Sumitra Ganesh +1Hallucination in Language ModelsLLM Reliability

    J.P. Morgan AI Research

  20. Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation

    Feb 19, 2026Bogdan Kostić, Conor Fallon, Julian Risch +1Prompt SensitivityLLM Evaluation

    Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin

  21. Unexplored flaws in multiple-choice VQA make benchmarking unreliable

    Nov 27, 2025Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf +3Visual Question AnsweringPrompt Sensitivity

    Technical University of Munich · Volkswagen AG · Helmholtz AI +1

  22. Measuring Intent Comprehension in LLMs

    Jun 19, 2025Nadav Kunievsky, James A. EvansPrompt SensitivityLLM Evaluation

    Knowlesdge Lab, University of Chicago, Chicago, Illinois, USA · Knowledge Lab, Data Science Institute, Department of Sociology, University of Chicago, Chicago, Illinois, USA.

  23. Multidimensional Consistency Improves Reasoning in Language Models

    Mar 4, 2025Huiyuan Lai, Xiao Zhang, Malvina NissimReasoning Consistency in Language ModelsLLM Mathematical Reasoning

    CLCG, University of Groningen / The Netherlands

  24. PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?

    Feb 6, 2025Mennatullah SiamLarge Vision-Language ModelsVision-Language Grounding

  25. Algorithmic Collusion by Large Language Models

    Mar 31, 2024Sara Fish, Yannai A. Gonczarowski, Ran I. ShorrerPrompt SensitivityAlgorithmic Collusion

    †School of Engineering and Applied Sciences, Harvard University · ‡Department of Economics and Department of Computer Science, Harvard University · §Department of Economics, Penn State University