Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

Yuan An

cs.CYFeb 21, 2026

Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

Authors: Yuan An

Organizations: College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, USA

Abstract

Advances in large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape research activities remains limited. We report a mixed-methods pilot evaluation of an AI-orchestrated research workflow in which a human researcher coordinated multiple LLM-based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation. Using the generation and assessment of multiple-choice questions (MCQs) as a testbed, we collected 1,071 SAT Math MCQs and employed LLM agents to extract questions from PDFs, retrieve and convert open textbooks into structured representations, align each MCQ with relevant textbook content, generate new MCQs under specified difficulty and cognitive levels, and evaluate both original and generated MCQs using a 24-criterion quality framework. Across all evaluations, average MCQ quality was high. However, criterion-level analysis and equivalence testing show that generated MCQs are not fully comparable to expert-vetted baseline questions. Strict similarity (24/24 criteria equivalent) was never achieved. Persistent gaps concentrated in skill\ depth, cognitive engagement, difficulty calibration, and metadata alignment, while surface-level qualities, such as {grammar fluency}, {clarity options}, {no duplicates}, were consistently strong. Beyond MCQ outcomes, the study documents a labor shift. The researcher's work moved from authoring items'' toward {specification, orchestration, verification}, and {governance}. Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance constituted the primary activities. We discuss implications for the future of scientific work, including emerging AI research operations'' skills required for AI-empowered research pipelines.

Figures & tables

Figure 1: Overview of the end-to-end AI-orchestrated MCQ research workflow.

Figure 2: Example MCQ entry from the College Board question bank.

Figure 3: LLM-based MCQ generation, evaluation, and comparison

Set	Judge	#	Mean $\pm$ SD	High-quality	Needs review	Serious
				( $=$ 5)	( $\in$ [3, 4])	( $\in$ [1, 2])
SAT baseline	Gemini	1013	4.80 $\pm$ 0.27	88%	10%	2%
Gemini-generated	Gemini	1065	4.89 $\pm$ 0.15	93.4%	5.2%	1.4%
GPT-generated	Gemini	1071	4.82 $\pm$ 0.19	90%	8%	2%
SAT baseline	OpenAI	1016	4.64 $\pm$ 0.49	74.1%	24.5%	1.4%
Gemini-generated	OpenAI	1057	4.68 $\pm$ 0.43	76%	23%	1%

Table 1: Overall quality summary (mean over 24 criteria).

Category	Criterion	G $\times$ SAT	G $\times$ Gem	G $\times$ OAI	O $\times$ SAT	O $\times$ Gem	O $\times$ OAI
		$(n{=}1013)$	$(n{=}1065)$	$(n{=}1071)$	$(n{=}1016)$	$(n{=}1057)$	$(n{=}1058)$
Accuracy	factual_accuracy	4.98	4.98	4.94	4.98	4.98	4.98
	no_hallucinations	4.18	4.90	4.88	5.00	4.99	4.99
	correct_key_answer	4.98	4.94	4.94	4.95	4.94	4.95
	content_alignment	4.32	4.93	4.89	4.95	4.99	4.98
Clarity	clarity_question	4.99	4.98	4.97	4.87	4.94	4.95

Table 2: Mean evaluation score (1-5) per criterion across six judge-generator configurations. Criteria are grouped by thematic category. The three lowest-scoring criteria across all configurations are highlighted in bold . G = Gemini judge; O = OpenAI judge; SAT = original SAT items; Gem = Gemini-generated; OAI = OpenAI-generated.

Category	G $\times$ SAT	G $\times$ Gem	G $\times$ OAI	O $\times$ SAT	O $\times$ Gem	O $\times$ OAI
Accuracy & Grounding	4.48	4.94	4.91	4.91	4.98	4.98
Clarity & Presentation	4.95	4.99	4.98	4.91	4.96	4.96
Distractor & Answer Quality	4.73	4.88	4.85	4.54	4.64	4.66
Diff. & Cognitive Calibr.	4.78	4.96	4.84	4.60	4.70	4.62
Domain & Skill Alignment	4.70	4.73	4.64	4.52	4.56	4.52
Difficulty & Engagement	4.52	4.89	4.75	4.00	4.18	4.13

Table 3: Mean category scores across six judge-generator configurations. Categories are sorted by grand mean across all configurations. Scores below 4.5 are underlined.

		Gemini Judge		OpenAI Judge
Category	Criterion	Matched	Whole	Matched	Whole
Accuracy & Grounding	Factual accuracy	✓	✓	✓	✓
	No hallucinations	×	×	✓	✓
	Correct key answer	✓	✓	✓	✓
	Content alignment	×	×	✓	×
Clarity & Presentation	Stem clarity	✓	✓	×	×

Table 4: Per-criterion equivalence verdicts across judges and analysis tracks. A ✓ indicates that all three pairwise TOST comparisons fell within

\pm 0.2\,\textit{SD}

\alpha=0.05

; a × indicates at least one pair was not equivalent.

	Gemini Judge ( $n=986$ )				OpenAI Judge ( $n=966$ )
Category	$d_{z}$ G-O	$d_{z}$ G-S	$d_{z}$ O-S	Equiv.	$d_{z}$ G-O	$d_{z}$ G-S	$d_{z}$ O-S	Equiv.
Accuracy & Grounding	.057	.342	.240	×	.016	.044	.024	✓
Clarity & Presentation	.036	$-.022$	$-.055$	✓	$-.019$	.111	.136	✓
Distractor & Answer Qual.	.080	.077	$-.012$	✓	$-.060$	.157	.228	×
Difficulty & Engagement	.242	.349	.097	×	.090	.271	.178	×
Domain & Skill Alignment	.170	$-.017$	$-.141$	×	.101	.044	$-.047$	✓

Table 5: Category-level equivalence results (matched track). Each category is the mean of within-criterion

z

-scores for its constituent criteria. Cohen’s

d_{z}

is reported for each pairwise comparison; Equiv. indicates whether all three pairs pass TOST at

\delta=0.2\,\textit{SD}

\alpha=0.05

. Friedman’s

\chi^{2}

is the omnibus statistic.

	Gemini Judge				OpenAI Judge
Category	$g$ G-O	$g$ G-S	$g$ O-S	Equiv.	$g$ G-O	$g$ G-S	$g$ O-S	Equiv.
Accuracy & Grounding	.058	.472	.369	×	.012	.052	.036	✓
Clarity & Presentation	.036	$-.031$	$-.071$	✓	$-.036$	.137	.182	×
Distractor & Answer Qual.	.084	.087	$-.006$	✓	$-.080$	.209	.306	×
Difficulty & Engagement	.329	.497	.142	×	.118	.349	.232	×
Domain & Skill Alignment	.131	$-.040$	$-.179$	×	.117	.055	$-.069$	✓

Table 6: Category-level equivalence results (whole-set track, independent samples). Hedges’

g

is reported for each pairwise comparison. Welch’s ANOVA

F

provides the omnibus test.

	Gemini Judge		OpenAI Judge
Metric	Matched	Whole	Matched	Whole
Equivalent criteria (/24)	12	11	11	9
Equivalent categories (/6)	2	2	3	2
Track agreement (criteria)	23/24 (95.8%)		22/24 (91.7%)

Table 7: Summary of equivalence testing outcomes across all conditions. Track agreement indicates the proportion of criteria whose equivalence verdict is the same under both analysis tracks for a given judge.

Task Category	Concrete Examples	Autor et al. Type
Research Framing and Problem Definition	Day 1: defining the core hypothesis and comparison design; deciding baseline and success condition	Non-routine analytic
Data Acquisition and Formatting	Day 1-2: downloading SAT PDFs; extracting PDF content to JSON; extracting MCQs; manual source collection from OpenStax	Mixed: non-routine: prompting AIs, manually download PDFs; routine: generating code for file conversion
Pipeline Engineering and Operations	Day 3-8: building/testing scripts for OCR batching, JSON merging, chunking, embedding, retrieval mapping, segmented generation runs	Mixed: routine-heavy with non-routine troubleshooting
Evaluation Design and Statistical Analysis	Day 9-10: expanding criteria 15 $\rightarrow$ 24; implementing multi-set evaluation pipeline; aggregation and formal comparability analysis	Mixed: non-routine: analytics; routine: generate the analytic programs
Human Quality Control and Interpretation	Day 4-10: validating scripts/results, spot-checking outputs, agreeing/disagreeing with model ratings, interpreting final findings	Non-routine: interaction/judgment

Table 8: Classification of research tasks in the AI-orchestrated workflow, mapped to the task framework of Autor et al. (2003) .

Task Category	Hours	%
Research Framing and Problem Definition	8	10%
Data Acquisition and Formatting	10	12.5%
Pipeline Engineering and Operations	42	52.5%
Evaluation Design and Statistical Analysis	12	15%
Human Quality Control and Interpretation	8	10%
Total	80	100%

Table 9: Approximate researcher time allocation across task categories (total

\approx

80 hours over 10 days).

Metric	Traditional PhD Timeline	AI-Orchestrated Workflow
Estimated Duration	$\sim$ 6 Months	$\sim$ 10 Days
Labor Intensity	Full-time	$\sim$ 8 Hours/Day
Variable API Cost	—	$\sim$ $62 (Mistral + OpenAI + Gemini)

Table 10: Illustrative contrast of research resource requirements.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Description	Source	N
SAT baseline	Expert-vetted SAT Math MCQs (text extracted from PDFs; incomplete items filtered)	College Board SAT Suite Question Bank PDFs	$\sim$ 1,017
Gemini-generated	MCQs generated from open textbook chunks under domain, skill, difficulty, and cognitive-level constraints	LLM-based generator (Gemini)	$\sim$ 1,065
GPT-generated	MCQs generated from open textbook chunks under the same constraints	LLM-based generator (GPT)	$\sim$ 1,071

Appendix

Table 11: Datasets

Criterion	Group	Short definition (paraphrased)
factual_accuracy	Accuracy & Grounding	Stem/options/key are mathematically correct as written.
no_hallucinations	Accuracy & Grounding	Does not invent missing context; no unsupported claims.
correct_key_answer	Accuracy & Grounding	The labeled correct option is truly correct.
content_alignment	Accuracy & Grounding	Item targets the intended concept from the mapped chunk.
clarity_question	Clarity & Presentation	Stem is complete, readable, and unambiguous.
clarity_options	Clarity & Presentation	Options are parallel, readable, and distinguishable.

Appendix

Table 12: Evaluation Criteria

Multiple-Choice Question Generation

See all 8 papers

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
Jul 6, 2026Yufeng WangScientific Hypothesis Generation Large Language Model-Guided Scientific Discovery
Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design
Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4AI for Science Benchmark Design
Westlake University · Zhejiang University · Nanjing University of Science and Technology +1
Sci $Π$ vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
Oct 15, 2026Youran Sun, Xingyu Ren, Chugang Yi +15LLM Agent Evaluation AI Agent Benchmarks
University of Maryland, College Park · The Chinese University of Hong Kong · Gradient Network +3

Jul 6, 2026J/K move · Enter open · S save

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

Yufeng Wang

Scientific Hypothesis Generation Large Language Model-Guided Scientific Discovery

Open paper AskPDF arXiv

LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.

Abstract

Multiple-Choice Question Generation

Explore similar work

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

SciΠΠΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Sci $Π$ vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents