Multiple-Choice Question Answering

Also known as MCQA

Multiple-choice question answering (MCQA) studies how models use knowledge, reasoning, and evidence to choose among candidate answers, across settings such as MMLU, HellaSwag, ScienceQA, and medical or multilingual exams. Common approaches include chain-of-thought prompting, likelihood and option scoring, retrieval-augmented generation, fine-tuning with classification heads, and methods that compare or filter options. Researchers also test efficiency and internal decision processes, including compact decision heads, executable reasoning scaffolds, and interventions on model representations. A central tension is that accuracy can hide sensitivity to option order, wording, popularity, and candidate-set size, as well as failures to abstain when choices are invalid.

Latest papers

43 papers9 papers in the last four weeks, up 13% on the four weeks before. 0.1% of all new papers.

Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
Oct 26, 2026Mingyang Song, Mao Zheng, Zhiyuan Ma +1Language Model Bias Evaluation Position Bias
Foundation Model Department, Tencent, China
Investigating Length Bias and Robustness in LVLMs for Multiple-Choice Question Answering
Oct 17, 2026Md. Atabuzzaman, Hani Alomari, Chris ThomasVLM Robustness Multiple-Choice Question Answering
Department of Computer Science, Virginia Tech
One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation
Oct 14, 2026Hyunjong Ok, Minjae Park, Jaeho LeeInput Order Sensitivity in Language Models Language Model Bias Evaluation
Pohang University of Science and Technology (POSTECH)
LEV: Thinking Fast and Deep with Looped Decision Models
Oct 14, 2026Dingjie Song, Zhiling Yan, Yuxuan Zhang +1Efficient Language Model Inference Adaptive Computation
Lehigh University · University of British Columbia · Vector Institute
Consider, Then Choose: Order-Invariant Decisions in System One Models
Oct 13, 2026Jianguo Huang, Yao Long Teng, Yanchen Deng +1Input Order Sensitivity in Language Models Multiple-Choice Question Answering
Nanyang Technological University, Singapore
FunDM: Cheap & Label-Free Decision Models from Frozen Language Models
Oct 12, 2026Ujjawal K. PanchalLLM Decision-Making Efficient Language Model Inference
Fundamental AI
Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks
Oct 8, 2026Xing Li, Qingcheng Chang, Jinzhong Ning +5LLM Evaluation Multiple-Choice Question Answering
Dalian Maritime University · Dalian University of Technology
Penalty-Framed No-Valid-Option MCQA: Analyzing LLM Abstention under Invalid Choices
Oct 6, 2026Jinhyeok Kim, Hye-Young JungLLM Evaluation Multiple-Choice Question Answering
Applied Artificial Intelligence Hanyang University, Republic of Korea · Mathematical Data Science Hanyang University, Republic of Korea
AnyJev Technical Report
Sep 30, 2026Jiamu Zhang, Tianze Yang, Yucheng Shi +6Multiple-Choice Question Answering Language Model Calibration
Nokia, Sunnyvale, CA, USA · Tencent Hunyuan · The Hong Kong Polytechnic University
Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling
Sep 28, 2026Mohamed Eltahir, Abobaker Ahmed, Nawaf Barebood +3Multiple-Choice Question Answering Benchmark Validity
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England
Overwhelmed by Choice: Studying LLM Decision Making at Scale
Sep 26, 2026Yu-Chi Lin, Aryan Seth, Anshul Aravind +4LLM Evaluation LLM Decision-Making
University of California, Los Angeles · University of Cincinnati
Likelihood Ranking doesn't Scale Like Prompting in LLMs
Sep 24, 2026Alessandro Bondielli, Lucia Passaro, Davide Bacciu +1LLM Evaluation Language Model Generation Evaluation
CoLingLab, Department of Philology, Literature and Linguistics, University of Pisa · Department of Computer Science, University of Pisa
Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
Sep 24, 2026Zeyan Li, Siyuan Qiu, Jianfeng XuVLM Evaluation Language Model Generation Evaluation
Shanghai Jiao Tong University
Option-Aware Retrieval and Task-Specific VLM Adaptation for Medical VQA
Sep 14, 2026Tristan Kirscher, Niklas C. Koser, Soren PirkVLM Adaptation Multiple-Choice Question Answering
ICube, Institut Strauss · CAU
Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation
Sep 14, 2026M. Ali BayramMultilingual Language Model Evaluation LLM Evaluation
Department of Computer Engineering, Yıldız Technical University İstanbul, Turkey
EAR: Entity-Aware Partitioning Approach for Retrieval-Augmented Generation Development
Sep 10, 2026Cenab Batu Bora, Oylum Alatlı, Sebnem Bora +1Document Chunking Retrieval-Augmented Generation
Georgia Institute of Technology, Atlanta, GA, USA · Ege University, Izmir, Turkey
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Sep 8, 2026Manikandan Ravikiran, Siddharth VohraMultiple-Choice Question Answering LLM Reliability
Indian Institute of Technology Mandi, India · Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA
StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions
Sep 1, 2026Chao Gao, Haijiang Liu, Qiyuan Li +3Prompt Sensitivity Multiple-Choice Question Answering
Department of Computer Science, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands · School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, China · Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, China
Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation
Aug 30, 2026Nishant Balepur, Paiheng Xu, Wei Ai +3LLM Evaluation Multiple-Choice Question Answering
University of Maryland · New York University · Nanyang Technological University
Large Language Models Systematically Favor Popular Options: Evidence and Mitigation Across MCQs
Aug 29, 2026Abdelrahman Abdallah, Mohammed Ali, Bhawna Piryani +2Popularity Bias Language Model Bias Evaluation
University of Innsbruck · Chungbuk National University
BengaliMCQ: Automatic Generation and Answer Prediction of Academic Multiple-Choice Questions in a Low-Resource Language
Aug 16, 2026Abu Tarabin Surzo, A. K. M. Nihalul Kabir, Sm Azmain Faysal +3Multiple-Choice Question Generation Multiple-Choice Question Answering
Department of Computer Science and Engineering, BRAC University, Kha 224 Pragati Sarani, Merul Badda, Dhaka, 1212, Bangladesh
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
Aug 15, 2026Volodymyr OvcharovLLM Evaluation Multiple-Choice Question Answering
LEX AI
From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options
Aug 13, 2026Obed Junias, Maria Leonor PachecoCompositional Reasoning Multiple-Choice Question Answering
University of Colorado Boulder
A, B, C, or Stable? Accuracy and Order Sensitivity Diverge in LLM Multiple-Choice Evaluation
Aug 12, 2026Karl Hanna, Chen FengLLM Evaluation Multiple-Choice Question Answering
Queen’s University Belfast
Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models
Aug 1, 2026Okan S. Coskun, Florian Rottach, Carsten Eickhoff +1Multiple-Choice Question Answering LLM Decision-Making
University of Tubingen · The University of Texas at Austin
Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval
Jul 24, 2026Justice Ayela, Kabir SahniFinance Multilingual Language Model Evaluation
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332
Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering
Jul 22, 2026Zhuohan Xie, Yuyang Dai, Rania Elbadry +18Multilingual Language Model Evaluation Multiple-Choice Question Answering
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria · University of Arizona, Tucson, United States +6
Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks
Jul 4, 2026Bhavesh Sood, Jaromir SavelkaFine-Tuning Multiple-Choice Question Answering
Carnegie Mellon University
Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions
Jul 1, 2026César Guerra-Solano, Xiang Lorraine LiLLM Evaluation Language Model Generation Evaluation
Department of Computer Science, University of Pittsburgh
Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval
Jun 9, 2026João Maria Janeiro, Mathurin Videau, Andrea Caciolai +3LLM Evaluation Multiple-Choice Question Answering
FAIR at Meta · Sorbonne Université, CNRS, ISIR, F-75005 Paris, France · Criteo AI Lab, Paris, France

Oct 26, 2026J/K move · Enter open · S save

Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?

Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou

Foundation Model Department, Tencent, China

Language Model Bias Evaluation Position Bias Multiple-Choice Question Answering

Open paper AskPDF arXiv

Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27.3--28.5% relative to the original order, with first failures continuing into the final presentation. Reordering loses substantially more initial successes than fixed-order repetition, a gap that also appears in highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.