Prompt Sensitivity

Ask about this topic

Overview

Prompt sensitivity studies how models respond to changes such as paraphrases, word order, option labels, examples, schemas, and contextual framing, including when the intended meaning stays the same. Researchers measure output flips, score variance, consistency, and ranking changes across controlled prompt variants, and use prompt aggregation, robustness training, and internal-state analyses to explain or reduce these effects. The issue spans language, vision-language, clinical, safety, and benchmark settings, where prompt choices can affect both correctness and behavior. A central tension is that greater consistency does not guarantee correctness or safety, and methods that help in one model or task may fail in another.

Latest papers

CardsList

115 papers18 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

  1. The Butterfly Defects: Tracing Cooperation in LLM Agent Networks

    Oct 23, 2026Maria Sampani, Maria Lymperaiou, Angeliki Dimitriou +3Multi-LLM CollaborationMulti-Agent Coordination

    National Technical University of Athens

  2. Prompt Sensitivity in Tool-Augmented Model Comparisons

    Oct 22, 2026Mateusz Modrzejewski, Tomasz RadzikowskiLLM EvaluationPrompt Sensitivity

    Institute of Computer Science, Warsaw University of Technology

  3. What Pairwise Evaluation Misses: Reward Scores across Prompts

    Oct 22, 2026Mateusz Modrzejewski, Tomasz RadzikowskiReward Model EvaluationPairwise Preference Evaluation

    Warsaw University of Technology

  4. Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

    Oct 21, 2026Louise YorkeLLM-as-a-JudgeLLM Evaluation

    University College London, London, UK

  5. How Prompt Style Affects AI Accuracy and Speed in Competitive Programming

    Oct 21, 2026Saad B. Alarifi, Mohammed A. AlamLLM PromptingPrompt Sensitivity

    Riyadh Schools Riyadh, Saudi Arabia · Department of Computer Science Northwestern University Evanston, IL, USA

  6. LLMs Prompted for Legal Context Refuse More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

    Oct 19, 2026Anastasiia Kucherenko, Francois Brouchoud, Dimitri Percia David +1LLM Refusal BehaviorPrompt Sensitivity

    IEM, HEG, HES-SO Valais-Wallis Sierre, Switzerland

  7. Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

    Oct 18, 2026Kazem Faghih, Yize Cheng, Shoumik Saha +3LLM EvaluationLLM Reliability

    Department of Computer Science, University of Maryland, College Park

  8. Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

    Oct 15, 2026Zhiyuan Wang, Han Wang, Shuo ZhangInput Order Sensitivity in Language ModelsMixture-of-Experts Inference

    Independent Researcher

  9. Cross-Prompt Inference Mitigates Rare-Class Suppression in Imbalanced VLM Finetuning

    Oct 15, 2026Emanuel Ben Baruch, Sivan Biham, Alon Shoshan +2VLM InferenceVLM Adaptation

    Amazon

  10. Representation Redistribution in Sparse Autoencoders

    Oct 15, 2026Ziye Mei, Haotian MaSparse AutoencodersPrompt Sensitivity

    Department of Computer Sciences, University of Wisconsin–Madison

  11. Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution

    Oct 13, 2026Giulio Corsi, Maryam Abo-Tabik, Yael Benn +4Automated GradingLLM-as-a-Judge

    Centre for the Future of Intelligence, University of Cambridge, Cambridge, UK · Department of Computer Science, University of Lancashire, Preston, UK · School of Psychology, Manchester Metropolitan University, Manchester, UK +3

  12. Learning a Fact Is Not Learning How to Retrieve It

    Oct 2, 2026Chaemin Jang, Jihee Kim, Dongman LeeFactual Knowledge in Language ModelsFactual QA

    Korea Advanced Institute of Science and Technology (KAIST)

  13. Labels Override Definitions in Jev-Style Typed Decision Models

    Oct 1, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramLanguage Model Bias EvaluationPrompt Sensitivity

    University of Southern California (USC), USA

  14. Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

    Oct 1, 2026Noy Sternlicht, Simra Shahid, Peter Jansen +3Prompt SensitivityLLM-as-a-Judge

    Hebrew University of Jerusalem · Allen Institute for AI · Microsoft +2

  15. Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

    Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt SensitivityLLM Evaluation

    Technion – Israel Institute of Technology

  16. Semifactual Credit-Augmented Policy Optimization

    Sep 30, 2026Junshu Pan, Zhizhang Fu, Shulin Huang +5Prompt SensitivityRL for Language Model Reasoning

    Zhejiang University · Westlake University · Shanghai Innovation Institute +1

  17. Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models

    Sep 28, 2026Yusheng Zhou, Eleanor Lin, David JurgensPrompt SensitivityLanguage Model Generation Evaluation

    Electrical Engineering and Computer Science Department University of Michigan Ann Arbor, MI 48105, USA

  18. Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks

    Sep 28, 2026Panagiota Kyriazi, Eleni Kasoura, Prokopis ProkopidisMultilingual Language Model EvaluationPrompt Sensitivity

    Institute for Language and Speech Processing / Athena RC

  19. Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark

    Sep 26, 2026Atharv Gupta, Akshat Jindal, Lavanya Nigam +1LLM SycophancyLanguage Model Generation Evaluation

    Indian Institute of Technology Roorkee

  20. Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

    Sep 24, 2026Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2Prompt SensitivityLLM-as-a-Judge

    KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST +1

  21. Memory vs. Context? Influential Factors of Factual Recall in Language Models

    Sep 21, 2026Guilhem Fouilhé, Nicholas Asher, Philippe MullerPrompt SensitivityMemorization in Language Models

    IRIT ANITI · IRIT CNRS · IRIT Université de Toulouse

  22. Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

    Sep 17, 2026Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5VLM RobustnessPrompt Sensitivity

    Sapienza University of Rome · Université Côte d’Azur · CNRS +3

  23. AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content

    Sep 15, 2026David Ababio Awuni, Rose-Mary Owusuaa Mensah Gyening, Elvis Gyasi OwusuMultilingual Language Model EvaluationPrompt Sensitivity

  24. Another Blueprint In The Wall: How to Ask Frontier AI Like a Kid?

    Sep 13, 2026Afshin KhadangiPrompt SensitivityLLM Prompting

    University of Luxembourg

  25. Sweet Talkers: How Query Formulation Shapes Sycophancy in Romantic Relationship Advice

    Sep 12, 2026Helena Choi, Edric Castel Hao, Karl Bautista +3LLM SycophancyFraming Effects in Language Models

    Ateneo de Manila SHS · Analog Devices, Inc.

  26. IchthyoNoma: Nomenclature and Context Sensitivity of Zero-Shot Biological Vision--Language Models for Bangladeshi Freshwater Fish Recognition

    Sep 3, 2026Nazim-E-Alam, Tarek Rahman, Md Kishor MorolVLM RobustnessPrompt Sensitivity

    American International University Bangladesh · United International University · ELITE Research Lab, Queens, New York, USA

  27. Analysis of Prompt Engineering for Drug Toxicity Prediction

    Sep 3, 2026Mia MacGregor, Aakash Welgamage Don, Mark BartlettPrompt SensitivityLLM Prompting

    School of Computing, Engineering & Technology, Robert Gordon University, Aberdeen, UK. · School of Pharmacy, Applied Sciences & Public Health, Robert Gordon University, Aberdeen, UK.

  28. Prompt-Robust Language Models: Which Training Strategies Work?

    Sep 1, 2026Frederic Sadrieh, Michal ŠtefánikPrompt SensitivityLLM Fine-Tuning

    Center for Information and Language Processing, LMU Munich · R&D Centre for Large Language Models, National Institute of Informatics, Japan

  29. StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions

    Sep 1, 2026Chao Gao, Haijiang Liu, Qiyuan Li +3Prompt SensitivityMultiple-Choice Question Answering

    Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands · School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China · Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, China

  30. The Fragility of Jailbreak Robustness Across Operational States

    Aug 31, 2026Yuna Park, Hwang Youn Kim, Yujin Kim +3Prompt SensitivityLanguage Model Safety Evaluation

    Yonsei University · Korea Institute of Science and Technology (KIST) · Korea University +1