Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

Xu Guo; Qiming Ge; Jian Tong; Kedi Chen; Jin Zhang; Xiaogui Yang; Xuan Gao; Haijun Lv

cs.CLMar 13, 2026

Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

Authors: Xu Guo, Qiming Ge, Jian Tong, Kedi Chen, Jin Zhang, Xiaogui Yang, Xuan Gao, Haijun Lv,

Organizations: Shanghai AI Laboratory · Shanghai Innovation Institute · Fudan University

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.

Figures & tables

Figure 1: The impact of distractor design on RLVR.

Figure 2: Relationship between average normalized

z

score and option-count gap

\Delta

. Note that data points are slightly offset horizontally for clarity.

Training Configuration	Llama-3.1-8B	Qwen2-7B
Train on 2-way	52.44	46.66
2 $\to$ 10 (Self)	56.12	50.71
2 $\to$ 10 (Qwen2.5-32B)	55.91	51.01
2 $\to$ 10 (GPT-OSS-120B)	56.26	50.88
Train on 4-way	56.19	48.57
4 $\to$ 10 (Self)	56.72	50.60

Table 1: Effect of option expansion on RLVR performance. All results are evaluated on the mcq10 test set and reported as accuracy (%). Bold and Underlined indicate the best and second-best results.

Llama-3.1-8B		Qwen2-7B
Configuration	Acc	Configuration	Acc
mcq2-random	52.44	mcq2-random	46.66
w/o Llama-3.1-8B	49.36	w/o Qwen2-7B	39.87
w/ Llama-3.1-8B	53.22	w/ Qwen2-7B	46.98
w/ Qwen2-7B	52.25	w/ Llama-3.1-8B	46.94
w/ DS-V3.1	51.13	w/ DS-V3.1	46.41

Table 2: Impact of distractor strength on RLVR performance (%). We compare random distractors (mcq2-random), selecting the weakest distractor ( w/o , lowest

\hat{s}_{j}

), or selecting the strongest distractor ( w/ , highest

\hat{s}_{j}

) as estimated by a reference model. DS-V3.1 denotes DeepSeek-V3.1. Bold and Underlined indicate the best and second-best results.

Figure 3: Training dynamics analysis: response length (top row) and solve-all ratio (bottom row) for Llama (left column) and Qwen (right column) across distractor settings. Removing strong distractors (w/o distractor) leads to rapid saturation and shorter responses.

Figure 4: The framework of Iterative Distractor Curation (IDC).

Dataset	Qwen2-7B						Llama-3.1-8B
Dataset	Base	Orig.	Direct	Filter	Rewrite	IDC	Base	Orig.	Direct	Filter	Rewrite	IDC
MedQA	47.21	49.25	48.70	48.00	48.39	49.80	59.07	62.14	65.20	65.04	64.41	66.14
SuperGPQA (Clinical)	25.62	25.53	26.11	26.60	26.27	27.91	33.25	33.58	35.39	35.39	35.39	35.71
MMLU-Pro (Health)	44.38	46.21	47.07	46.09	45.72	49.51	58.56	57.46	57.95	59.17	58.19	60.76
MedXpertQA	10.24	10.98	10.29	10.86	10.24	11.88	14.29	16.29	15.47	16.45	15.67	19.31
PubMedQA	69.50	72.50	72.00	72.90	71.20	74.00	78.00	77.80	78.30	78.50	78.80	77.60

Table 3: Downstream performance on medical datasets. Base : Base instruct model. Orig. : Standard RLVR on original MCQs. Direct : Direct conversion to short-answer. Filter : Filtering non-convertible questions. Rewrite : Model-based conversion. IDC : Our Iterative Distractor Curation method. All results reported as accuracy (%).

Dataset	Qwen2-7B						Llama-3.1-8B
Dataset	Base	Self	Qwen-32B	OSS	DS-V3.1	Llama-70B	Base	Self	Qwen-32B	OSS	DS-V3.1	Llama-70B
MedQA	47.21	47.29	49.80	48.70	49.41	49.88	59.07	66.61	66.14	65.75	67.87	67.09
SuperGPQA	25.62	27.75	27.91	28.49	27.75	27.34	33.25	36.45	35.71	38.51	34.81	36.45
MMLU-Pro	44.38	49.88	49.51	46.21	47.07	47.31	58.56	61.00	60.76	59.17	60.88	61.00
MedXpertQA	10.24	12.45	11.88	13.14	11.80	12.49	14.29	20.29	19.31	19.27	17.96	18.04
PubMedQA	69.50	74.60	74.00	71.60	71.50	72.50	78.00	78.10	77.60	76.80	77.90	77.00

Table 4: Ablation on distractor generators. We compare models trained with distractors generated by the model itself (“Self”) versus stronger external models. “Base” denotes the original instruct model without RLVR. Qwen-32B: Qwen2.5-32B-Instruct; OSS: GPT-OSS-120B; DS-V3.1: DeepSeek-V3.1; Llama-70B: Llama-3.1-70B-Instruct.

Figure 5: Performance gains and distractor dynamics across rejection sampling rounds.

Stage	Wall-Clock Time
Distractor Generation (per round)	$\sim$ 15–20 min
Strength Evaluation (per round)	$<$ 5 min
Full IDC Pipeline (7 rounds)	$\sim$ 1.5 h
RLVR Training	18–20 h
IDC Overhead / Total	$<$ 10%

Table 5: Computational cost breakdown on 8

\times

H200 GPUs (MedQA,

\sim

10k samples).

Factor	Setting	$\mathbf{n}$	Spurious Rate
Factor	Setting	$\mathbf{n}$	Before	After
Quantity	mcq2 $\rightarrow$ mcq10	400	0.241	0.095
Quality	w/o $\rightarrow$ w/ IDC	200	0.260	0.110

Table 6: Spurious reward rates before and after varying option design on MMLU-Pro (row 1) and MedQA (row 2).

n

: annotated response pairs.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Table 7: Cross-option generalization results (Accuracy %). Bold and Underlined denote the best and second-best results in each column (excluding the mix strategy). Background color intensity represents normalized score

z

, with darker colors indicating better performance.

Dataset	Base	w/ Origin	Self	DS-V3.1	OSS	Qwen2.5-32B	Llama-70B
MedQA	69.76	72.74	73.92	74.71	74.86	73.13	74.23
SuperGPQA	37.19	38.83	40.07	39.16	39.66	38.92	40.48
MMLU-Pro	63.94	64.79	65.53	65.16	65.77	64.55	64.30
MedXpertQA	14.41	16.41	18.20	18.04	18.69	18.65	17.96
PubMedQA	77.90	76.80	78.00	78.20	77.30	78.50	78.60
Average	52.64	53.91	55.14	55.05	55.26	54.75	55.11

Appendix

Table 8: Ablation study on distractor generators for Qwen2.5-14B-Instruct. The Base column shows the performance of the original instruct model without RLVR training. The w/ Origin column shows training with original distractors from the dataset. OSS: GPT-OSS-120B; DS-V3.1: DeepSeek-V3.1.

Dataset	Train Qwen2-7B		Train Llama-3.1-8B
Dataset	Est: Self	Est: Llama	Est: Self	Est: Qwen
MedQA	49.80	49.41	66.14	67.71
SuperGPQA	27.91	27.42	35.71	35.63
MMLU-Pro	49.51	48.04	60.76	60.39
MedXpertQA	11.88	11.59	19.31	18.45
PubMedQA	74.00	73.30	77.60	78.50

Appendix

Table 9: Ablation on the strength estimator. We fix the generator and compare different models for estimating distractor strength. “Self” indicates that the estimator is the same model as the one being trained (e.g., using Qwen2-7B to estimate for Qwen2-7B), while other names indicate cross-model estimation.

Panel of Figure 6 — Figure 6: Option-label distributions under origin vs. mcq2 vs. mcq10 training. Both models show A/B bias after mcq2 training, a near-uniform distribution after mcq10 , and origin in between.

Dataset	Qwen2-7B		Llama-3.1-8B
Dataset	Orig.	IDC	Orig.	IDC
MedQA	47.16 $\pm$ 0.29	49.72 $\pm$ 0.44	59.01 $\pm$ 0.38	66.02 $\pm$ 0.47
SuperGPQA	25.53 $\pm$ 0.40	27.85 $\pm$ 0.38	33.24 $\pm$ 0.36	35.65 $\pm$ 0.43
MMLU-Pro	44.35 $\pm$ 0.32	49.43 $\pm$ 0.49	58.53 $\pm$ 0.39	60.70 $\pm$ 0.48
MedXpertQA	10.20 $\pm$ 0.33	11.82 $\pm$ 0.37	14.25 $\pm$ 0.43	19.22 $\pm$ 0.43
PubMedQA	69.39 $\pm$ 0.40	73.91 $\pm$ 0.42	77.92 $\pm$ 0.44	77.58 $\pm$ 0.48

Appendix

Table 10: Multi-seed results (mean

\pm

std over 3 independent runs) for the standard RLVR baseline (Orig.) and IDC. IDC provides stable and consistent gains across runs.

Model	Train	Test-2	Test-4	Test-6	Test-8	Test-10	Avg
Llama-3.1-8B	mix	80.01	70.21	62.86	59.46	57.56	66.02
	Best single (mcq6)	79.57	69.58	64.07	59.21	57.87	66.06
	Single-strategy Avg	78.71	69.36	63.12	58.40	56.66	65.25
Qwen2-7B	mix	71.29	64.57	56.99	51.94	49.01	58.76
	Best single (mcq4)	72.15	65.15	58.22	51.92	48.57	59.20
	Single-strategy Avg	71.46	63.79	57.24	51.95	48.67	58.62

Appendix

Table 11: Mixed-option training results on MMLU-Pro. “Single-strategy Avg” denotes the average over five single-option-count training runs (mcq2, mcq4, mcq6, mcq8, mcq10). The mix strategy outperforms the single-strategy average and provides robust generalization without prior knowledge of the test distribution.

Setting	Accuracy (mean $\pm$ std)
Base (Qwen2.5-14B-Instruct)	61.72 $\pm$ 1.51%
Orig. (standard RLVR)	68.32 $\pm$ 0.57%
IDC	71.05 $\pm$ 1.14%

Appendix

Table 12: IDC results on the law subset of MMLU-Pro using Qwen2.5-14B-Instruct (3-seed average).

Model	mcq2-test	mcq10-test
Qwen2.5-14B (base)	81.88	62.63
+ mcq2-train	86.13	62.92
+ mcq10-train	83.79	67.94

Appendix

Table 13: Option-count mismatch persists in Qwen2.5-14B-Instruct. The mcq2-trained model excels on mcq2-test but shows minimal improvement on mcq10-test, and vice versa.

N_{d}=\sum_{k=1}^{K}\mathbb{I}[a_{k}=d,\,a_{k}\neq o^{*}],\quad N_{\text{err}}=\sum_{k=1}^{K}\mathbb{I}[a_{k}\neq o^{*}],

Appendix

Algorithm 1 Iterative Distractor Curation (IDC)

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Jan 16, 2026Lecheng Yan, Ruizhe Li, Guanhua Chen +5Reward Hacking Memorization in Language Models
Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen +5
Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR
Sep 8, 2026Youngjun Yu, Sanghwan Jang, Hwanjo YuRL for Language Model Reasoning RL Exploration
Pohang University of Science and Technology (POSTECH)
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
Mar 21, 2026Kaiyuan Li, Jing-Cheng Pang, Yang YuLanguage Model Reasoning Evaluation RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology & School of Artificial Intelligence, Nanjing University, China · Polixir.ai, China · Huawei Technologies Co., Ltd., China

Jan 16, 2026J/K move · Enter open · S save

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li+4

Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen+5

Reward Hacking Memorization in Language Models LLM Interpretability RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.

Abstract

RL for Language Model Reasoning

Explore similar work

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution