Overview

Prompt sensitivity studies how models respond to changes such as paraphrases, word order, option labels, examples, schemas, and contextual framing, including when the intended meaning stays the same. Researchers measure output flips, score variance, consistency, and ranking changes across controlled prompt variants, and use prompt aggregation, robustness training, and internal-state analyses to explain or reduce these effects. The issue spans language, vision-language, clinical, safety, and benchmark settings, where prompt choices can affect both correctness and behavior. A central tension is that greater consistency does not guarantee correctness or safety, and methods that help in one model or task may fail in another.

Latest papers

CardsList

121 papers9 papers in the last four weeks, down 40% on the four weeks before. 0.1% of all new papers.

  1. Measuring Definition Sensitivity in LLM Annotation of Social Science Concepts

    Nov 9, 2026Bogdan Mamaev, Qizhou Wang, Christopher Leckie +1LLM-Assisted AnnotationPrompt Sensitivity

  2. The Brittle Compass: Navigating LLM Prompt Sensitivity in Slovak Migration Media Discourse

    Oct 31, 2026Samuel Harvan, Jaroslav Kopčan, Marek Šuppa +1Prompt SensitivityLLM Prompting

  3. Measured or Real? When Model-Derived Measurements Can Be Corrected or Compared

    Oct 27, 2026Yifeng Li, Ziang NiLLM EvaluationLLM Reliability

  4. The Butterfly Defects: Tracing Cooperation in LLM Agent Networks

    Oct 23, 2026Maria Sampani, Maria Lymperaiou, Angeliki Dimitriou +3Multi-LLM CollaborationMulti-Agent Coordination

  5. Prompt Sensitivity in Tool-Augmented Model Comparisons

    Oct 22, 2026Mateusz Modrzejewski, Tomasz RadzikowskiLLM EvaluationPrompt Sensitivity

  6. Poolwise: Evaluating Reward Models for Selection across Prompts

    Oct 22, 2026Mateusz Modrzejewski, Tomasz RadzikowskiReward Model EvaluationPairwise Preference Evaluation

  7. How Prompt Style Affects AI Accuracy and Speed in Competitive Programming

    Oct 21, 2026Saad B. Alarifi, Mohammed A. AlamLLM PromptingPrompt Sensitivity

  8. LLMs Prompted for Legal Context Refuse More: Overrefusal from Small On-Premises LLMs in Criminal Legal Context

    Oct 19, 2026Anastasiia Kucherenko, Francois Brouchoud, Dimitri Percia David +1LLM Refusal BehaviorPrompt Sensitivity

  9. Revealing what benchmark scores hide: Jointly measuring the breadth and reliability of LLM capabilities using accessibility profiles

    Oct 18, 2026Kazem Faghih, Yize Cheng, Shoumik Saha +3LLM EvaluationLLM Reliability

  10. Representation Redistribution in Sparse Autoencoders

    Oct 15, 2026Ziye Mei, Haotian MaSparse AutoencodersPrompt Sensitivity

  11. Machine--human agreement in marking undergraduate assignments is specific to model, prompt and institution

    Oct 13, 2026Giulio Corsi, Maryam Abo-Tabik, Yael Benn +4Automated GradingLLM-as-a-Judge

  12. Learning a Fact Is Not Learning How to Retrieve It

    Oct 2, 2026Chaemin Jang, Jihee Kim, Dongman LeeKnowledge Retrieval in Language ModelsFactual Knowledge in Language Models

  13. Labels Override Definitions in Jev-Style Typed Decision Models

    Oct 1, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramTyped Decision ModelsLanguage Model Bias Evaluation

  14. Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

    Oct 1, 2026Noy Sternlicht, Simra Shahid, Peter Jansen +3Prompt SensitivityLLM-as-a-Judge

  15. Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

    Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt SensitivityLLM Evaluation

  16. Semifactual Credit-Augmented Policy Optimization

    Sep 30, 2026Junshu Pan, Zhizhang Fu, Shulin Huang +5Prompt SensitivityRL for Language Model Reasoning

  17. Lost in Translation: Measuring the Effect of Non-Native English on End User Performance of Large Language Models

    Sep 28, 2026Yusheng Zhou, Eleanor Lin, David JurgensPrompt SensitivityLanguage Model Generation Evaluation

  18. Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks

    Sep 28, 2026Panagiota Kyriazi, Eleni Kasoura, Prokopis ProkopidisMultilingual Language Model EvaluationPrompt Sensitivity

  19. Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark

    Sep 26, 2026Atharv Gupta, Akshat Jindal, Lavanya Nigam +1LLM SycophancyLanguage Model Generation Evaluation

  20. Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

    Sep 24, 2026Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2Prompt SensitivityLLM-as-a-Judge

  21. Memory vs. Context? Influential Factors of Factual Recall in Language Models

    Sep 21, 2026Guilhem Fouilhé, Nicholas Asher, Philippe MullerPrompt SensitivityMemorization in Language Models

  22. Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

    Sep 17, 2026Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5VLM RobustnessPrompt Sensitivity

  23. Sweet Talkers: How Query Formulation Shapes Sycophancy in Romantic Relationship Advice

    Sep 12, 2026Helena Choi, Edric Castel Hao, Karl Bautista +3LLM SycophancyFraming Effects in Language Models

  24. IchthyoNoma: Nomenclature and Context Sensitivity of Zero-Shot Biological Vision--Language Models for Bangladeshi Freshwater Fish Recognition

    Sep 3, 2026Nazim-E-Alam, Tarek Rahman, Md Kishor MorolVLM RobustnessPrompt Sensitivity

  25. Analysis of Prompt Engineering for Drug Toxicity Prediction

    Sep 3, 2026Mia MacGregor, Aakash Welgamage Don, Mark BartlettPrompt SensitivityLLM Prompting