SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation

Jiahao Zhao; Feng Jiang; Shaowei Qin; Zhonghui Zhang; Junhao Liu; Guibing Guo; Hamid Alinejad-Rokny; Min Yang

cs.AIFeb 26, 2026

SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation

Authors: Jiahao Zhao, Feng Jiang, Shaowei Qin, Zhonghui Zhang, Junhao Liu, Guibing Guo, Hamid Alinejad-Rokny, Min Yang

Organizations: Software College, Northeastern University, China · Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China · University of California, Irvine, CA, USA ·

Abstract

Large language models (LLMs) are increasingly applied in scientific research, offering new capabilities for knowledge discovery and reasoning. In single-cell biology, however, evaluation practices for both general and specialized LLMs remain inadequate: existing benchmarks are fragmented across tasks, adopt formats such as multiple-choice classification that diverge from real-world usage, and rely on metrics lacking interpretability and biological grounding. We present SC-ARENA, a natural language evaluation framework tailored to single-cell foundation models. SC-ARENA formalizes a virtual cell abstraction that unifies evaluation targets by representing both intrinsic attributes and gene-level interactions. Within this paradigm, we define five natural language tasks (cell type annotation, captioning, generation, perturbation prediction, and scientific QA) that probe core reasoning capabilities in cellular biology. To overcome the limitations of brittle string-matching metrics, we introduce knowledge-augmented evaluation, which incorporates external ontologies, marker databases, and scientific literature to support biologically faithful and interpretable judgments. Experiments and analysis across both general-purpose and domain-specialized LLMs demonstrate that (i) under the Virtual Cell unified evaluation paradigm, current models achieve uneven performance on biologically complex tasks, particularly those demanding mechanistic or causal understanding; and (ii) our knowledge-augmented evaluation framework ensures biological correctness, provides interpretable, evidence-grounded rationales, and achieves high discriminative capacity, overcoming the brittleness and opacity of conventional metrics. SC-Arena thus provides a unified and interpretable framework for assessing LLMs in single-cell biology, pointing toward the development of biology-aligned, generalizable foundation models.

Figures & tables

Framework	Evaluation Paradigm	Evaluation Format	Evaluation Metrics
C2S-Scale	Cell Sentence	Prompted Completion	Lexical & Statistical Matching
Cell-o1	Reasoning Agent	Constrained Selection	Classification Matching
SOAR	Single-Task Agent	QA-based Classification	Surface String Matching
CELLVERSE	Multi-omics Cell Sentence	Multiple Choice Questions	Classification Matching
SC-ARENA (Ours)	Virtual Cell	Open-Ended Natural Language QA	Knowledge-Grounded Matching

Table 1: Comparison of existing single-cell LLM evaluation frameworks.

Figure 1: Overview of the SC-ARENA framework.

Task	Data Source	#Samples	Format (Input $\rightarrow$ Output)
CTA	CELLxGENE	608	cell sentence (expression) $\rightarrow$ ontology label (CL)
CC	CELLxGENE	608	cell sentence (expression) $\rightarrow$ natural-language caption
CG	CELLxGENE	608	Ontology/cell-type name $\rightarrow$ cell sentence (expression)
PP	Norman; Adamson	138	Control & perturbed cell sentences $+$ perturbation spec $\rightarrow$ (i) up/down DEGs; (ii) post-perturbation cell sentence
SQA	PubMed	254	Question $\rightarrow$ natural-language answer $+$ evidence rationale

Table 2: Summary of benchmark datasets in SC-ARENA .

General-purpose Models
Model	CTA	CG	CC	PP	SQA	Total
Qwen2.5-7B	12.61	45.98	51.05	28.84	64.09	202.57
Qwen2.5-14B	25.89	51.74	56.05	34.78	66.37	236.06
Qwen2.5-32B	26.78	50.95	55.46	36.23	66.77	238.06
Qwen3-8B	21.45	50.53	57.20	37.39	72.83	254.45
Qwen3-14B	29.17	15.50	60.88	32.89	72.20	210.64

Table 3: Performance of different models across five tasks (with Total Score). Bold = 1st place in column, Underline = 2nd place in column. Non-numeric entries ( —) are excluded from ranking.

Panel of Figure 2 — Figure 2: (a) Relationship between prediction score and ontology distance (Spearman $\rho=0.6212$ , p <0.001); (b) Example scoring responses using external knowledge.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	BERTScore	BLEU-1	BLEU-2	ROUGE-1	ROUGE-2	ROUGE-L	METEOR
Qwen2.5-7B	80.20	0.24	0.02	0.12	0.02	0.12	0.07
Qwen2.5-14B	83.45	0.68	0.10	0.22	0.05	0.22	0.13
Qwen2.5-32B	82.93	0.94	0.08	0.23	0.04	0.23	0.07
Qwen3-8B	80.98	0.07	0.01	0.17	0.04	0.17	0.09
Qwen3-14B	80.86	18.38	4.54	0.21	0.06	0.21	0.12
Qwen3-32B	84.18	22.22	7.09	0.23	0.07	0.23	0.16

Appendix

Table 4: Performance comparison of different models across traditional metrics in cell type annotation.

Model	BERTScore	BLEU-1	BLEU-2	ROUGE-1	ROUGE-2	ROUGE-L	METEOR
Qwen2.5-7B	81.72	7.25	0.74	0.12	0.01	0.09	0.16
Qwen2.5-14B	82.29	8.44	0.89	0.13	0.01	0.09	0.17
Qwen2.5-32B	82.22	7.59	0.88	0.12	0.01	0.09	0.17
Qwen3-8B	82.16	3.49	0.32	0.12	0.01	0.09	0.15
Qwen3-14B	81.91	8.73	0.82	0.13	0.01	0.09	0.16
Qwen3-32B	81.87	8.07	0.73	0.13	0.01	0.09	0.16

Appendix

Table 5: Performance comparison of different models across traditional metrics in cell captioning.

Model	BERTScore	BLEU-1	ROUGE-1	ROUGE-L	METEOR
Qwen2.5-7B	75.17	0.12	0.00	0.00	0.00
Qwen2.5-14B	77.01	0.36	0.01	0.01	0.00
Qwen2.5-32B	77.28	0.72	0.02	0.01	0.01
Qwen3-8B	75.17	0.08	0.00	0.00	0.00
Qwen3-14B	71.69	0.30	0.00	0.00	0.00
Qwen3-32B	77.27	0.33	0.01	0.01	0.01

Appendix

Table 6: Performance comparison of different models across traditional metrics in cell generation.

Model	BERTScore	BLEU-1	BLEU-2	ROUGE-1	ROUGE-2	ROUGE-L	METEOR
Qwen2.5-7B	87.40	88.39	29.92	0.56	0.08	0.21	0.34
Qwen2.5-14B	90.39	90.95	23.45	0.77	0.11	0.28	0.45
Qwen2.5-32B	90.92	91.99	22.01	0.83	0.12	0.30	0.48
Qwen3-8B	85.71	88.84	27.27	0.51	0.09	0.21	0.30
Qwen3-14B	88.19	86.11	24.98	0.62	0.10	0.24	0.37
Qwen3-32B	89.06	91.19	27.01	0.65	0.11	0.26	0.37

Appendix

Table 7: Performance comparison of different models across traditional metrics in perturbation prediction.

Model	BERTScore	BLEU-1	BLEU-2	ROUGE-1	ROUGE-2	ROUGE-L	METEOR
Qwen2.5-7B	88.51	43.22	14.10	0.38	0.13	0.23	0.25
Qwen2.5-14B	88.91	48.02	16.20	0.40	0.15	0.25	0.27
Qwen2.5-32B	89.04	52.43	17.42	0.40	0.15	0.25	0.26
Qwen3-8B	87.81	29.20	9.08	0.39	0.13	0.22	0.31
Qwen3-14B	85.46	32.65	9.73	0.39	0.12	0.22	0.30
Qwen3-32B	74.81	30.46	8.70	0.33	0.10	0.18	0.26

Appendix

Table 8: Performance comparison of different models across traditional metrics in ScienceQA.

Model	BLEU-1	BLEU-2	CTA_Score
C2S-Pythia-410m	23.73	17.14	47.34
C2S-Scale-Pythia-1b	13.66	8.29	41.68
DeepSeek-R1	41.55	19.55	40.81
Kimi-K2	31.60	10.91	40.00

Appendix

Table 9: BLEU scores for C2S models on the Cell Type Annotation (CTA) task. Low BLEU scores—despite strong biological performance—support that results are not due to data leakage or memorization.

Task	Validation Metric	Spearman $\rho$	$p$ -value
CTA	Ontology path distance	0.6212	$<10^{-3}$
CG	% correct CellMarker genes	0.6669	$<10^{-3}$
PP	DEG cosine similarity	0.6057	$<10^{-3}$
CC	Expert preference alignment	0.8394	$<10^{-3}$
SQA	Expert preference alignment	0.8199	$<10^{-3}$

Appendix

Table 10: Reliability validation of the knowledge-augmented evaluator across SC-ARENA tasks. Metrics reflect correlation between evaluator scores and external validity measures (Spearman

\rho

and

p

-value). For open-ended tasks, we report expert agreement rate and correlation with evaluator rankings.

Figure 3: Radar-plot comparison of representative general-purpose models across the five SC-ARENA tasks: cell type annotation, perturbation prediction, cell generation, cell captioning, and scientific QA. The visualization highlights the uneven distribution of gains: while models such as Kimi-K2 and DeepSeek-R1 excel in captioning and generation, Qwen3-32B performs comparatively better in perturbation prediction. The radar plot provides a task-level perspective that complements aggregate scores and illustrates persistent challenges in mechanistic reasoning.

Figure 4: Distribution of ontology path length to root for predicted cell types across models. The x-axis shows binned intervals of the average path length to the ontology root, using left-closed, right-open notation [a,b), and the y-axis indicates the number of predicted cell types falling into each interval. Shorter path lengths indicate closer alignment with the ontology hierarchy and thus more specific predictions.

Robustness Test	Spearman $\rho$	Kendall’s $\tau$	Cosine Similarity
Cross-model evaluator consistency	0.6701	0.5719	0.8873
Insensitivity to response length	0.0730	0.0578	–
Robustness to evaluation stochasticity (CTA task)	0.9070	0.8354	0.9468

Appendix

Table 11: Quantitative metrics for robustness validation of the knowledge-augmented evaluator.

Figure 5: Cell Type Annotation Answer Generation Prompt.

Figure 6: Cell Captioning Answer Generation Prompt.

Figure 7: Cell Generation Answer Generation Prompt.

Figure 8: Perturbation Prediction Answer Generation Prompt.

Figure 9: Scientific QA Answer Generation Prompt.

Figure 10: Cell Type Annotation Score Generation Prompt.

Figure 11: Cell Captioning Score Generation Prompt.

Figure 12: Cell Generation Score Generation Prompt.

Figure 13: Perturbation Prediction Score Generation Prompt.

Figure 14: Scientific QA Score Generation Prompt.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology
Jun 25, 2026Ian Diks, Zhen Yang, Arjun Banerjee +2AI Agents for Scientific Discovery Long-Horizon Agent Evaluation
LatchBio, San Francisco, CA, USA
MAT-Cell: A Multi-Agent Tree-Structured Reasoning Framework for Batch-Level Single-Cell Annotation
Apr 7, 2026Yehui Yang, Zelin Zang, Xienan Zheng +9LLM-Assisted Annotation Single-Cell Transcriptomics
Westlake University, Hangzhou, China · Shenzhen University of Advanced Technology, Shenzhen, China · Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China +1
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
May 19, 2026Yiyang Gu, Junwei Yang, Junyu Luo +15AI for Science LLM Evaluation
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5

Jun 25, 2026J/K move · Enter open · S save

scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology

Ian Diks, Zhen Yang, Arjun Banerjee, Tim Proctor+1

LatchBio, San Francisco, CA, USA

AI Agents for Scientific Discovery Long-Horizon Agent Evaluation Single-Cell Transcriptomics Agent Benchmarks

Open paper AskPDF arXiv

Single-cell studies require analysts to convert raw measurements into specific biological claims through multi-step workflows and integration of metadata, assay context, and auxiliary evidence. Existing AI-biology benchmarks largely measure broad knowledge, executable workflows, or local analysis steps. We introduce scBench-Long, a benchmark for long-horizon single-cell biology in which agents must recover scientific conclusions from raw or near-raw data without prescribed methods. The benchmark contains 21 evaluations spanning melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human--monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Tasks cover paired scRNA/TCR sequencing, RNA and chromatin profiling, cross-species transcriptomics, combinatorial scRNA-seq, single-nucleus RNA-seq, immune repertoires, ortholog maps, ligand--receptor resources, and validation evidence. Candidate claims are reproduced, reviewed, and converted into controlled answer vocabularies with deterministic grading and trajectory rubrics. Across 1,068 completed trajectories, the strongest model--harness pair passes 16/63 runs (25.4%). scBench-Long evaluates whether agents can move beyond local analysis steps and make complex scientific claims that are supported by single-cell data.

Abstract

LLM Evaluation

Explore similar work

scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology

MAT-Cell: A Multi-Agent Tree-Structured Reasoning Framework for Batch-Level Single-Cell Annotation

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models