Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

Yuan An

cs.CYFeb 21, 2026

Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

Authors: Yuan An

Organizations: College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, USA

Abstract

Advances in large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape research activities remains limited. We report a mixed-methods pilot evaluation of an AI-orchestrated research workflow in which a human researcher coordinated multiple LLM-based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation. Using the generation and assessment of multiple-choice questions (MCQs) as a testbed, we collected 1,071 SAT Math MCQs and employed LLM agents to extract questions from PDFs, retrieve and convert open textbooks into structured representations, align each MCQ with relevant textbook content, generate new MCQs under specified difficulty and cognitive levels, and evaluate both original and generated MCQs using a 24-criterion quality framework. Across all evaluations, average MCQ quality was high. However, criterion-level analysis and equivalence testing show that generated MCQs are not fully comparable to expert-vetted baseline questions. Strict similarity (24/24 criteria equivalent) was never achieved. Persistent gaps concentrated in skill\ depth, cognitive engagement, difficulty calibration, and metadata alignment, while surface-level qualities, such as {grammar fluency}, {clarity options}, {no duplicates}, were consistently strong. Beyond MCQ outcomes, the study documents a labor shift. The researcher's work moved from authoring items'' toward {specification, orchestration, verification}, and {governance}. Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance constituted the primary activities. We discuss implications for the future of scientific work, including emerging AI research operations'' skills required for AI-empowered research pipelines.

Figures & tables

Figure 1: Overview of the end-to-end AI-orchestrated MCQ research workflow.

Figure 2: Example MCQ entry from the College Board question bank.

Figure 3: LLM-based MCQ generation, evaluation, and comparison

Set	Judge	#	Mean $\pm$ SD	High-quality	Needs review	Serious
				( $=$ 5)	( $\in$ [3, 4])	( $\in$ [1, 2])
SAT baseline	Gemini	1013	4.80 $\pm$ 0.27	88%	10%	2%
Gemini-generated	Gemini	1065	4.89 $\pm$ 0.15	93.4%	5.2%	1.4%
GPT-generated	Gemini	1071	4.82 $\pm$ 0.19	90%	8%	2%
SAT baseline	OpenAI	1016	4.64 $\pm$ 0.49	74.1%	24.5%	1.4%
Gemini-generated	OpenAI	1057	4.68 $\pm$ 0.43	76%	23%	1%

Table 1: Overall quality summary (mean over 24 criteria).

Category	Criterion	G $\times$ SAT	G $\times$ Gem	G $\times$ OAI	O $\times$ SAT	O $\times$ Gem	O $\times$ OAI
		$(n{=}1013)$	$(n{=}1065)$	$(n{=}1071)$	$(n{=}1016)$	$(n{=}1057)$	$(n{=}1058)$
Accuracy	factual_accuracy	4.98	4.98	4.94	4.98	4.98	4.98
	no_hallucinations	4.18	4.90	4.88	5.00	4.99	4.99
	correct_key_answer	4.98	4.94	4.94	4.95	4.94	4.95
	content_alignment	4.32	4.93	4.89	4.95	4.99	4.98
Clarity	clarity_question	4.99	4.98	4.97	4.87	4.94	4.95

Table 2: Mean evaluation score (1-5) per criterion across six judge-generator configurations. Criteria are grouped by thematic category. The three lowest-scoring criteria across all configurations are highlighted in bold . G = Gemini judge; O = OpenAI judge; SAT = original SAT items; Gem = Gemini-generated; OAI = OpenAI-generated.

Category	G $\times$ SAT	G $\times$ Gem	G $\times$ OAI	O $\times$ SAT	O $\times$ Gem	O $\times$ OAI
Accuracy & Grounding	4.48	4.94	4.91	4.91	4.98	4.98
Clarity & Presentation	4.95	4.99	4.98	4.91	4.96	4.96
Distractor & Answer Quality	4.73	4.88	4.85	4.54	4.64	4.66
Diff. & Cognitive Calibr.	4.78	4.96	4.84	4.60	4.70	4.62
Domain & Skill Alignment	4.70	4.73	4.64	4.52	4.56	4.52
Difficulty & Engagement	4.52	4.89	4.75	4.00	4.18	4.13

Table 3: Mean category scores across six judge-generator configurations. Categories are sorted by grand mean across all configurations. Scores below 4.5 are underlined.

		Gemini Judge		OpenAI Judge
Category	Criterion	Matched	Whole	Matched	Whole
Accuracy & Grounding	Factual accuracy	✓	✓	✓	✓
	No hallucinations	×	×	✓	✓
	Correct key answer	✓	✓	✓	✓
	Content alignment	×	×	✓	×
Clarity & Presentation	Stem clarity	✓	✓	×	×

Table 4: Per-criterion equivalence verdicts across judges and analysis tracks. A ✓ indicates that all three pairwise TOST comparisons fell within

\pm 0.2\,\textit{SD}

\alpha=0.05

; a × indicates at least one pair was not equivalent.

	Gemini Judge ( $n=986$ )				OpenAI Judge ( $n=966$ )
Category	$d_{z}$ G-O	$d_{z}$ G-S	$d_{z}$ O-S	Equiv.	$d_{z}$ G-O	$d_{z}$ G-S	$d_{z}$ O-S	Equiv.
Accuracy & Grounding	.057	.342	.240	×	.016	.044	.024	✓
Clarity & Presentation	.036	$-.022$	$-.055$	✓	$-.019$	.111	.136	✓
Distractor & Answer Qual.	.080	.077	$-.012$	✓	$-.060$	.157	.228	×
Difficulty & Engagement	.242	.349	.097	×	.090	.271	.178	×
Domain & Skill Alignment	.170	$-.017$	$-.141$	×	.101	.044	$-.047$	✓

Table 5: Category-level equivalence results (matched track). Each category is the mean of within-criterion

z

-scores for its constituent criteria. Cohen’s

d_{z}

is reported for each pairwise comparison; Equiv. indicates whether all three pairs pass TOST at

\delta=0.2\,\textit{SD}

\alpha=0.05

. Friedman’s

\chi^{2}

is the omnibus statistic.

	Gemini Judge				OpenAI Judge
Category	$g$ G-O	$g$ G-S	$g$ O-S	Equiv.	$g$ G-O	$g$ G-S	$g$ O-S	Equiv.
Accuracy & Grounding	.058	.472	.369	×	.012	.052	.036	✓
Clarity & Presentation	.036	$-.031$	$-.071$	✓	$-.036$	.137	.182	×
Distractor & Answer Qual.	.084	.087	$-.006$	✓	$-.080$	.209	.306	×
Difficulty & Engagement	.329	.497	.142	×	.118	.349	.232	×
Domain & Skill Alignment	.131	$-.040$	$-.179$	×	.117	.055	$-.069$	✓

Table 6: Category-level equivalence results (whole-set track, independent samples). Hedges’

g

is reported for each pairwise comparison. Welch’s ANOVA

F

provides the omnibus test.

	Gemini Judge		OpenAI Judge
Metric	Matched	Whole	Matched	Whole
Equivalent criteria (/24)	12	11	11	9
Equivalent categories (/6)	2	2	3	2
Track agreement (criteria)	23/24 (95.8%)		22/24 (91.7%)

Table 7: Summary of equivalence testing outcomes across all conditions. Track agreement indicates the proportion of criteria whose equivalence verdict is the same under both analysis tracks for a given judge.

Task Category	Concrete Examples	Autor et al. Type
Research Framing and Problem Definition	Day 1: defining the core hypothesis and comparison design; deciding baseline and success condition	Non-routine analytic
Data Acquisition and Formatting	Day 1-2: downloading SAT PDFs; extracting PDF content to JSON; extracting MCQs; manual source collection from OpenStax	Mixed: non-routine: prompting AIs, manually download PDFs; routine: generating code for file conversion
Pipeline Engineering and Operations	Day 3-8: building/testing scripts for OCR batching, JSON merging, chunking, embedding, retrieval mapping, segmented generation runs	Mixed: routine-heavy with non-routine troubleshooting
Evaluation Design and Statistical Analysis	Day 9-10: expanding criteria 15 $\rightarrow$ 24; implementing multi-set evaluation pipeline; aggregation and formal comparability analysis	Mixed: non-routine: analytics; routine: generate the analytic programs
Human Quality Control and Interpretation	Day 4-10: validating scripts/results, spot-checking outputs, agreeing/disagreeing with model ratings, interpreting final findings	Non-routine: interaction/judgment

Table 8: Classification of research tasks in the AI-orchestrated workflow, mapped to the task framework of Autor et al. (2003) .

Task Category	Hours	%
Research Framing and Problem Definition	8	10%
Data Acquisition and Formatting	10	12.5%
Pipeline Engineering and Operations	42	52.5%
Evaluation Design and Statistical Analysis	12	15%
Human Quality Control and Interpretation	8	10%
Total	80	100%

Table 9: Approximate researcher time allocation across task categories (total

\approx

80 hours over 10 days).

Metric	Traditional PhD Timeline	AI-Orchestrated Workflow
Estimated Duration	$\sim$ 6 Months	$\sim$ 10 Days
Labor Intensity	Full-time	$\sim$ 8 Hours/Day
Variable API Cost	—	$\sim$ $62 (Mistral + OpenAI + Gemini)

Table 10: Illustrative contrast of research resource requirements.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Description	Source	N
SAT baseline	Expert-vetted SAT Math MCQs (text extracted from PDFs; incomplete items filtered)	College Board SAT Suite Question Bank PDFs	$\sim$ 1,017
Gemini-generated	MCQs generated from open textbook chunks under domain, skill, difficulty, and cognitive-level constraints	LLM-based generator (Gemini)	$\sim$ 1,065
GPT-generated	MCQs generated from open textbook chunks under the same constraints	LLM-based generator (GPT)	$\sim$ 1,071

Appendix

Table 11: Datasets

Criterion	Group	Short definition (paraphrased)
factual_accuracy	Accuracy & Grounding	Stem/options/key are mathematically correct as written.
no_hallucinations	Accuracy & Grounding	Does not invent missing context; no unsupported claims.
correct_key_answer	Accuracy & Grounding	The labeled correct option is truly correct.
content_alignment	Accuracy & Grounding	Item targets the intended concept from the mapped chunk.
clarity_question	Clarity & Presentation	Stem is complete, readable, and unambiguous.
clarity_options	Clarity & Presentation	Options are parallel, readable, and distinguishable.

Appendix

Table 12: Evaluation Criteria

Multiple-Choice Question Generation

See all 8 papers

Jul 6, 2026

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.

Yufeng Wang

Scientific Hypothesis Generation Large Language Model-Guided Scientific Discovery

Ask PDF

Aug 4, 2026

Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

AI for Research (AI4Research) leverages AI to automate and improve scientific workflows. While experimental design is a critical stage of the research process, prior work has focused primarily on code implementation and execution, overlooking the importance of this stage, and no benchmark exists to evaluate AI's ability to conduct systematic experiment design. To bridge this gap, we propose SCOPE, a Scientific COmprehensive Planning Evaluation Benchmark constructed from 300 high-quality latest papers across 19 research domains from top-tier venues (e.g., ICML, NeurIPS, and ICLR),evaluating LLMs on two dimensions: High-Level planning completeness (main, ablation, and analysis experiments) and Low-Level configuration accuracy and rationality (datasets, baselines, and metrics). Benchmarking reveals three findings: (1) most LLMs cannot directly design high-quality experiments; (2) all LLMs exhibit a performance bottleneck in low-level configuration; and (3) search mode does not improve design quality. Furthermore, to address these challenges, we propose OptED, a novel agentic workflow to optimize LLM-based experimental design, that enhances LLM-based experimental planning through stage isolation, tool augmentation, and rule-based constraints, effectively alleviating the configuration bottleneck.

Zejun Liu, Jian Wu, Ru Peng +4

Westlake University · Zhejiang University · Nanjing University of Science and Technology +1

AI for Science Benchmark Design AI Agent Benchmarks Large Language Model-Guided Scientific Discovery Autonomous Experimentation

Ask PDF

Oct 15, 2026

Sci $Π$ vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgment. Evaluating scientific judgment is difficult because research decisions are open-ended and depend on project context. We construct Sci

Π

vot, a 248-case benchmark of scientific judgment based on human corrections. The benchmark draws on records from six months of research in which senior researchers ran LLM agents on 26 projects across six scientific fields and repeatedly intervened to correct the agents' research judgments. In each case, an agent chooses between the original LLM plan and the human-corrected alternative, using only the project record available before the correction as context. We compare LLM and human performance and examine the effects of reasoning effort, access to the project record, and reconsideration. At medium reasoning effort with the project record, models systematically prefer the original LLM plan over the human-corrected alternative. Four of five frontier models perform below chance on the selected cases. The highest accuracy in the project condition across the tested models and reasoning effort settings is 63.5%, compared with 73.4% for fresh researchers. At medium effort, every model achieves lower accuracy with the project record than without it, and reconsideration does not consistently improve overall accuracy. Even with a human-corrected alternative available, frontier models struggle with the selected research decisions, while Sci

Π

vot supports systematic evaluation of scientific judgment for LLM agents.

Youran Sun, Xingyu Ren, Chugang Yi +15

University of Maryland, College Park · The Chinese University of Hong Kong · Gradient Network +3

LLM Agent Evaluation AI Agent Benchmarks Scientific Reasoning in Language Models Autonomous Research Agents

Ask PDF

Abstract

Multiple-Choice Question Generation

Explore similar work

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

SciΠΠΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents

Sci $Π$ vot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents