Optimizing Coverage and Difficulty in Reinforcement Learning for Quiz Composition

Ricardo Pedro Querido Andrade Silva; Nassim Bouarour; Dina Fettache; Sarab Boussouar; Noha Ibrahim; Sihem Amer-Yahia

cs.LGMar 29, 2026

Optimizing Coverage and Difficulty in Reinforcement Learning for Quiz Composition

Authors: Ricardo Pedro Querido Andrade Silva, Nassim Bouarour, Dina Fettache, Sarab Boussouar, Noha Ibrahim, Sihem Amer-Yahia

Organizations: CNRS, Univ. Grenoble Alpes Grenoble, France · Université Grenoble Alpes Grenoble, France

Abstract

Quiz design is a tedious process that teachers undertake to evaluate the acquisition of knowledge by students. Our goal in this paper is to automate quiz composition from a set of multiple choice questions (MCQs). We formalize a generic sequential decision-making problem with the goal of training an agent to compose a quiz that meets the desired topic coverage and difficulty levels. We investigate DQN, SARSA and A2C/A3C, three reinforcement learning solutions to solve our problem. We run extensive experiments on synthetic and real datasets that study the ability of RL to land on the best quiz. Our results reveal subtle differences in agent behavior and in transfer learning with different data distributions and teacher goals. This was supported by our user study, paving the way for automating various teachers' pedagogical goals.

Figures & tables

Figure 1. MCQs used to generate quizzes.

Dataset	Description	#MCQs	#Topics	#Diff. levels
MeD	generated using MedGemma 9B 1 1 1 For anonymity reasons, we omit citing the original paper. It will be added after acceptance. based on medical documents from the UNESS https://entrainement-ecn.uness.fr/ platform for training medical professionals.	1500+	111 topics from which we sampled 10 topics, e.g., Cardiology, Rhumatology.	The difficulty level of each MCQ has a numerical scale within $[1,5]$ , 1 being the easiest.
Math ( 2 )	collected from MathE https://mathe.ipb.pt/ an e-learning platform for enhancing mathematical skills in higher education and supporting autonomous MCQ-based learning.	1900+	15 topics from which we sampled 10 topics, e.g., Linear Algebra, Probabilities.	The difficulty level of each MCQ on a increasing scale within $[1,6]$ .

Table 1. Description of real datasets.

Dataset	Description	Example quiz representation
Uniform	Uniformly drawn MCQs	$T_{C}$ = <0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1> $T_{D}$ = <0.2, 0.2, 0.2, 0.2, 0.2>
BiasedTopic	Biased topics and uniform difficulties	$T_{C}$ = <0, 0, 0, 0, 0, 0.5, 0, 0, 0.5, 0> $T_{D}$ = <0.2, 0.2, 0.2, 0.2, 0.2>
BiasedLevel	Biased difficulties and uniform topics	$T_{C}$ = <0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1> $T_{D}$ = <0.5, 0, 0, 0, 0.5>

Table 2. Description of Synthetic datasets.

Figure 2. Evolution of max $Q$ -function across training iterations of DQN on target $T_{\mathit{uniform}}$ for different datasets and $\alpha$ values.

Figure 3. Evolution of learned actions per episode during DQN training on target $T_{\mathit{uniform}}$ for Uniform dataset.

Algos	$\alpha$	Uniform	BiasedTopic	BiasedLevel
DQN	0	0.896	0.903	0.888
	0.25	0.877	0.880	0.885
	0.5	0.870	0.870	0.862
	0.75	0.868	0.860	0.860
	1	0.877	0.851	0.869
Oracle	0	0.998	0.997	1.000

Table 3. Average similarity with target

T_{\mathit{uniform}}

for all Synthetic datasets.

Figure 4. Evolution of learned actions per episode during DQN training on target $T_{\mathit{bias}}$ for Uniform dataset.

Figure 5. Examples of agent trajectories in Uniform dataset using UMAP 2D projection.

Data	Algos	$\alpha$
Data	Algos	0	0.25	0.5	0.75	1
MeD	DQN	0.918	0.878	0.871	0.860	0.893
	SARSA	0.911	0.880	0.876	0.870	0.852
	A2C	0.907	0.877	0.869	0.861	0.886
	A3C	0.910	0.880	0.869	0.866	0.899
	Oracle	1.000	0.978	0.977	0.988	1.000

Table 4. Average similarity with target for all agents in real datasets when finding

T_{\mathit{uniform}}

Data	Algos	# Iterations	Time (sec)
MeD	DQN	20	0.04
	SARSA	29	0.07
	A2C	25	0.05
	A3C	17	0.04
	Oracle	10000	5.22
Math	DQN	31	0.07

Table 5. Average #iterations and time to find target

T_{\mathit{uniform}}

Figure 6. Evolution of number of iterations and similarity to target based on reward threshold for MeD dataset.

Training target	Inference target	Algos	Similarity w/target	#Iter
$T_{\mathit{uniform}}$	$T_{\mathit{bias}}$	DQN	0.756	100
		SARSA	0.791	100
		A2C	0.800	83
		A3C	0.775	95
		Oracle	0.888	10000
$T_{\mathit{bias}}$	$T_{\mathit{uniform}}$	DQN	0.866	24

Table 6. Average similarity and

\#

Iter (iterations) for transferring the learning of agents across targets on Math with

\alpha=0.5

Training data	Inference data	Algos	Similarity w/target	#Iter
MeD	Math	DQN	0.877	12
		SARSA	0.881	8
		A2C	0.865	16
		A3C	0.864	11
		Oracle	0.956	10000
Math	MeD	DQN	0.868	20

Table 7. Average similarity and

\#

Iter (Iterations) for transferring the learning of agents across datasets on

T_{\mathit{uniform}}

with

\alpha=0.5

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7. Projection of high-dimensional states in 2D with UMAP: a) Uniform dataset with $\alpha=0.5$ ; b) BiasedTopic dataset with $\alpha=1$ ; c) BiasedLevel dataset with $\alpha=0$ .

Figure 8. Evolution of topicMatch and diffMatch across training episodes of DQN on target

T_{\mathit{uniform}}

for Uniform dataset, with smoothing window of size 100.

Figure 9. Examples of agent trajectories in Uniform dataset using UMAP 2D projection following reward $\mathcal{R}_{1}$ (direct similarity reward); the colors of the agent trajectory represent the inference timeline, where time shifts from lighter to darker colors.

Figure 10. Examples of agent trajectories in Uniform dataset using UMAP 2D projection following reward $\mathcal{R}_{2}$ (progress-based reward).

Figure 11. Examples of agent trajectories in Uniform dataset using 2D projection with topicMatch and diffMatch following reward $\mathcal{R}_{1}$ (direct similarity reward).

Figure 12. Examples of agent trajectories in Uniform dataset using 2D projection with topicMatch and diffMatch following reward $\mathcal{R}_{2}$ (progress-based reward).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
Jun 2, 2026Rongzhi Zhang, Rui Feng, Zhihan Zhang +8Reinforcement Learning RL for Language Model Reasoning
Amazon · Georgia Institute of Technology
Playing 20 Question Game with Policy-Based Reinforcement Learning
Aug 23, 2018Huang Hu, Xianchao Wu, Bingfeng Luo +4Reinforcement Learning Game-Playing Agents
1Peking University, Beijing, China · 3Microsoft Development Co., Ltd, Tokyo, Japan · 2Microsoft Corporation, Beijing, China
Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation
Jun 8, 2026Gabriel Ortega, Abelino Jiménez, Séverin Lions +1Educational Assessment Item Difficulty Estimation
Centro de Investigación Avanzada en Educación (CIAE) Instituto de Estudios Avanzados en Educación (IE) Universidad de Chile. · Departamento de Evaluación Medición y Registro Educacional (DEMRE) Universidad de Chile. · Centro de Modelamiento Matemático (CMM) Universidad de Chile. +1

Jun 2, 2026J/K move · Enter open · S save

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang+7

Amazon · Georgia Institute of Technology

Reinforcement Learning RL for Language Model Reasoning Rubric-Based RL Rubric-Based Evaluation Reward Design for RL

Open paper AskPDF arXiv

Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed. We identify a structural bottleneck: rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. We present QUBRIC, a framework that co-designs queries and rubrics. Teacher-derived key points ground the rewriting of open-ended queries into scenario-based, evaluable questions. Contrastive rubric generation then turns teacher-policy gaps into query-level criteria, and learnability filtering retains only informative query-rubric pairs for GRPO training. QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results provide evidence that co-designing queries and rubrics can make rubric-based RL a practical complement to RLVR beyond strictly verifiable tasks.

Abstract

Reinforcement Learning

Explore similar work

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

Playing 20 Question Game with Policy-Based Reinforcement Learning

Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation