Prompt Sensitivity

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, down 10% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 68

All topics

Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Oct 1, 2026Noy Sternlicht, Simra Shahid, Peter Jansen +3Prompt Sensitivity LLM-as-a-Judge
Hebrew University of Jerusalem · Allen Institute for AI · Microsoft +2
Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs
Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt Sensitivity LLM Evaluation
Technion – Israel Institute of Technology
Semifactual Credit-Augmented Policy Optimization
Sep 30, 2026Junshu Pan, Zhizhang Fu, Shulin Huang +5Prompt Sensitivity RL for Language Model Reasoning
Zhejiang University · Westlake University · Shanghai Innovation Institute +1
Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks
Sep 28, 2026Panagiota Kyriazi, Eleni Kasoura, Prokopis ProkopidisMultilingual Language Model Evaluation Prompt Sensitivity
Institute for Language and Speech Processing / Athena RC
Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
Sep 24, 2026Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2Prompt Sensitivity LLM-as-a-Judge
KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST +1
Memory vs. Context? Influential Factors of Factual Recall in Language Models
Sep 21, 2026Guilhem Fouilhé, Nicholas Asher, Philippe MullerPrompt Sensitivity Memorization in Language Models
IRIT ANITI · IRIT CNRS · IRIT Université de Toulouse
Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models
Sep 17, 2026Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5VLM Robustness Prompt Sensitivity
Sapienza University of Rome · Université Côte d’Azur · CNRS +3
AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content
Sep 15, 2026David Ababio Awuni, Rose-Mary Owusuaa Mensah Gyening, Elvis Gyasi OwusuMultilingual Language Model Evaluation Prompt Sensitivity
Another Blueprint In The Wall: How to Ask Frontier AI Like a Kid?
Sep 13, 2026Afshin KhadangiPrompt Sensitivity LLM Prompting
University of Luxembourg
IchthyoNoma: Nomenclature and Context Sensitivity of Zero-Shot Biological Vision--Language Models for Bangladeshi Freshwater Fish Recognition
Sep 3, 2026Nazim-E-Alam, Tarek Rahman, Md Kishor MorolVLM Robustness Prompt Sensitivity
American International University Bangladesh · United International University · ELITE Research Lab, Queens, New York, USA
Analysis of Prompt Engineering for Drug Toxicity Prediction
Sep 3, 2026Mia MacGregor, Aakash Welgamage Don, Mark BartlettPrompt Sensitivity LLM Prompting
School of Computing, Engineering & Technology, Robert Gordon University, Aberdeen, UK. · School of Pharmacy, Applied Sciences & Public Health, Robert Gordon University, Aberdeen, UK.
Prompt-Robust Language Models: Which Training Strategies Work?
Sep 1, 2026Frederic Sadrieh, Michal ŠtefánikPrompt Sensitivity LLM Fine-Tuning
Center for Information and Language Processing, LMU Munich · R&D Centre for Large Language Models, National Institute of Informatics, Japan
StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions
Sep 1, 2026Chao Gao, Haijiang Liu, Qiyuan Li +3Prompt Sensitivity Multiple-Choice Question Answering
Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands · School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China · Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, China
The Fragility of Jailbreak Robustness Across Operational States
Aug 31, 2026Yuna Park, Hwang Youn Kim, Yujin Kim +3Prompt Sensitivity Language Model Safety Evaluation
Yonsei University · Korea Institute of Science and Technology (KIST) · Korea University +1
Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning
Aug 31, 2026Fukang Zhu, Binbin Zhao, Ruixiao Lin +3Prompt Sensitivity AI Coding Agents
1Zhejiang University · 2State Key Laboratory of Internet Architecture, Tsinghua University
A Probe Direction Is a Property of Its Prompt
Aug 13, 2026Valentin NoëlPrompt Sensitivity LLM Evaluation
Devoteam
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Aug 12, 2026Shailja Thakur, Sungeun An, Chad DeLuca +1Prompt Sensitivity LLM Evaluation
IBM Research India · IBM Research Almaden
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA
When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition
Aug 10, 2026Chencheng Zhu, Xiaoyang Li, Taotao CaiPrompt Sensitivity Weight-Space Analysis
UNSW Sydney · University of Southern Queensland
Your Prompt Is Not the Only Prompt: How Much Do LLMs Weight Structured-Output Schema Descriptions?
Aug 8, 2026Sin-Ying LinPrompt Sensitivity Structured Prompting
Independent Researcher
Confirming Our Biases? Evaluating the Capabilities, Risks, and Societal Impact of Large Language Models
Aug 7, 2026Mudar Adas, Polina Tsvilodub, Michael Franke +1Prompt Sensitivity LLM Sycophancy
Neuro-Cognitive Modeling Group, University of Tübingen, Sand 14, 72076 Tübingen, Germany · Department of Linguistics, University of Tübingen, Keplerstraße 2, 72074 Tübingen
The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections
Aug 4, 2026Marco Giunti, Fabrizia Giulia GaravagliaTransformer Inference Prompt Sensitivity
ALOPHIS – Applied Logic Philosophy and History of Science Università di Cagliari · Dip. di Pedagogia, Psicologia, Filosoﬁa Università di Cagliari
Passing Coarse Marginal Checks Can Be Cheap: Persona Mixtures and Imprecise Treatment-Response Estimates in an LLM Persona Panel
Aug 2, 2026Yohei NakajimaPrompt Sensitivity LLM Evaluation
Untapped Capital
Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity
Aug 1, 2026Yongxi Zhou, Junwei Yao, Yuanzhe Liu +4LLM Safety Benchmarks Prompt Sensitivity
1Northeastern University, Massachusetts, USA · 2Georgia Institute of Technology, Georgia, USA · University of Southern California, California, USA
Understanding the Impact of Linguistic Realization Choices on LLM Stance with Causal Tracing
Jul 22, 2026Langchen Huang, Sebastian Padó, Franziska WeeberPrompt Sensitivity LLM Interpretability
Institute for Natural Language Processing, University of Stuttgart
Production and Perception in LLMs: A Token Probability Approach
Jul 13, 2026Anna Marklová, Jiří Milička, Martina Vokáčová +1Prompt Sensitivity
Faculty of Arts, Charles University, Prague · Faculty of Mathematics and Physics, Charles University, Prague
A Decomposable Probe for Few-Step Diffusion Models: Prompt, Latent, and Score Selectivity across Backbone Families and Distillation Paradigms
Jul 3, 2026Patrick Mu HaojiePrompt Sensitivity Rectified Flow
Independent Researcher.
Understanding Evaluation Illusion in Diffusion Large Language Models
Jun 28, 2026Hengxiang Zhang, Jiaxi Ren, Renchunzi Xie +1Prompt Sensitivity LLM Evaluation
Department of Statistics and Data Science, Southern University of Science and Technology · Intelligent Computing Research Center, Great Bay Institute for Advanced Study, Great Bay University
Auditing Framing-Sensitive Behavioral Instability in Large Language Models for Mental Health Interactions
Jun 25, 2026Abla Bedoui, Ashley L. Greene, Mohammed CherkaouiPrompt Sensitivity LLM Auditing
School of computer science, digital engineering and AI, Long Island University, Brooklyn, NY, USA · Department of Psychology, Long Island University, Brooklyn, NY, USA
Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization
Jun 20, 2026Aman Goyal, Kshama Nitin Shah, Kemmannu Vineet Venkatesh RaoVLM Evaluation Zero-Shot Learning
Carnegie Mellon University · University of Michigan, Ann Arbor · Magna International

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld+2

Hebrew University of Jerusalem · Allen Institute for AI · Microsoft+2

Prompt Sensitivity LLM-as-a-Judge Automated Evaluation

Open paper AskPDF arXiv

Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We first build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.

Momentum

Related topics

Latest papers 68