SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Authors: Jiahao Zhao, Feng Jiang, Shaowei Qin, Zhonghui Zhang, Junhao Liu, Guibing Guo, Hamid Alinejad-Rokny, Min Yang
Organizations: Software College, Northeastern University, China · Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China · University of California, Irvine, CA, USA · School of Biomedical Engineering, UNSW Sydney, Sydney, Australia
Large language models (LLMs) are increasingly applied in scientific research, offering new capabilities for knowledge discovery and reasoning. In single-cell biology, however, evaluation practices for both general and specialized LLMs remain inadequate: existing benchmarks are fragmented across tasks, adopt formats such as multiple-choice classification that diverge from real-world usage, and rely on metrics lacking interpretability and biological grounding. We present SC-ARENA, a natural language evaluation framework tailored to single-cell foundation models. SC-ARENA formalizes a virtual cell abstraction that unifies evaluation targets by representing both intrinsic attributes and gene-level interactions. Within this paradigm, we define five natural language tasks (cell type annotation, captioning, generation, perturbation prediction, and scientific QA) that probe core reasoning capabilities in cellular biology. To overcome the limitations of brittle string-matching metrics, we introduce knowledge-augmented evaluation, which incorporates external ontologies, marker databases, and scientific literature to support biologically faithful and interpretable judgments. Experiments and analysis across both general-purpose and domain-specialized LLMs demonstrate that (i) under the Virtual Cell unified evaluation paradigm, current models achieve uneven performance on biologically complex tasks, particularly those demanding mechanistic or causal understanding; and (ii) our knowledge-augmented evaluation framework ensures biological correctness, provides interpretable, evidence-grounded rationales, and achieves high discriminative capacity, overcoming the brittleness and opacity of conventional metrics. SC-Arena thus provides a unified and interpretable framework for assessing LLMs in single-cell biology, pointing toward the development of biology-aligned, generalizable foundation models.
Figures & tables
Framework
Evaluation Paradigm
Evaluation Format
Evaluation Metrics
C2S-Scale
Cell Sentence
Prompted Completion
Lexical & Statistical Matching
Cell-o1
Reasoning Agent
Constrained Selection
Classification Matching
SOAR
Single-Task Agent
QA-based Classification
Surface String Matching
CELLVERSE
Multi-omics Cell Sentence
Multiple Choice Questions
Classification Matching
SC-ARENA (Ours)
Virtual Cell
Open-Ended Natural Language QA
Knowledge-Grounded Matching
Table 1: Comparison of existing single-cell LLM evaluation frameworks.
Table 2: Summary of benchmark datasets in SC-ARENA .
Model
CTA
CG
CC
PP
SQA
Total
General-purpose Models
Qwen2.5-7B
12.61
45.98
51.05
28.84
64.09
202.57
Qwen2.5-14B
25.89
51.74
56.05
34.78
66.37
236.06
Qwen2.5-32B
26.78
50.95
55.46
36.23
66.77
238.06
Qwen3-8B
21.45
50.53
57.20
37.39
72.83
254.45
Qwen3-14B
29.17
15.50
60.88
32.89
72.20
210.64
Table 3: Performance of different models across five tasks (with Total Score). Bold = 1st place in column, Underline = 2nd place in column. Non-numeric entries ( —) are excluded from ranking.
Figure 2: (a) Relationship between prediction score and ontology distance (Spearman ρ=0.6212 , p <0.001); (b) Example scoring responses using external knowledge.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
80.20
0.24
0.02
0.12
0.02
0.12
0.07
Qwen2.5-14B
83.45
0.68
0.10
0.22
0.05
0.22
0.13
Qwen2.5-32B
82.93
0.94
0.08
0.23
0.04
0.23
0.07
Qwen3-8B
80.98
0.07
0.01
0.17
0.04
0.17
0.09
Qwen3-14B
80.86
18.38
4.54
0.21
0.06
0.21
0.12
Qwen3-32B
84.18
22.22
7.09
0.23
0.07
0.23
0.16
Appendix
Table 4: Performance comparison of different models across traditional metrics in cell type annotation.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
81.72
7.25
0.74
0.12
0.01
0.09
0.16
Qwen2.5-14B
82.29
8.44
0.89
0.13
0.01
0.09
0.17
Qwen2.5-32B
82.22
7.59
0.88
0.12
0.01
0.09
0.17
Qwen3-8B
82.16
3.49
0.32
0.12
0.01
0.09
0.15
Qwen3-14B
81.91
8.73
0.82
0.13
0.01
0.09
0.16
Qwen3-32B
81.87
8.07
0.73
0.13
0.01
0.09
0.16
Appendix
Table 5: Performance comparison of different models across traditional metrics in cell captioning.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
75.17
0.12
0.00
0.00
0.00
0.00
0.00
Qwen2.5-14B
77.01
0.36
0.00
0.01
0.00
0.01
0.00
Qwen2.5-32B
77.28
0.72
0.00
0.02
0.00
0.01
0.01
Qwen3-8B
75.17
0.08
0.00
0.00
0.00
0.00
0.00
Qwen3-14B
71.69
0.30
0.00
0.00
0.00
0.00
0.00
Qwen3-32B
77.27
0.33
0.00
0.01
0.00
0.01
0.01
Appendix
Table 6: Performance comparison of different models across traditional metrics in cell generation.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
87.40
88.39
29.92
0.56
0.08
0.21
0.34
Qwen2.5-14B
90.39
90.95
23.45
0.77
0.11
0.28
0.45
Qwen2.5-32B
90.92
91.99
22.01
0.83
0.12
0.30
0.48
Qwen3-8B
85.71
88.84
27.27
0.51
0.09
0.21
0.30
Qwen3-14B
88.19
86.11
24.98
0.62
0.10
0.24
0.37
Qwen3-32B
89.06
91.19
27.01
0.65
0.11
0.26
0.37
Appendix
Table 7: Performance comparison of different models across traditional metrics in perturbation prediction.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
88.51
43.22
14.10
0.38
0.13
0.23
0.25
Qwen2.5-14B
88.91
48.02
16.20
0.40
0.15
0.25
0.27
Qwen2.5-32B
89.04
52.43
17.42
0.40
0.15
0.25
0.26
Qwen3-8B
87.81
29.20
9.08
0.39
0.13
0.22
0.31
Qwen3-14B
85.46
32.65
9.73
0.39
0.12
0.22
0.30
Qwen3-32B
74.81
30.46
8.70
0.33
0.10
0.18
0.26
Appendix
Table 8: Performance comparison of different models across traditional metrics in ScienceQA.
Model
BLEU-1
BLEU-2
CTA_Score
C2S-Pythia-410m
23.73
17.14
47.34
C2S-Scale-Pythia-1b
13.66
8.29
41.68
DeepSeek-R1
41.55
19.55
40.81
Kimi-K2
31.60
10.91
40.00
Appendix
Table 9: BLEU scores for C2S models on the Cell Type Annotation (CTA) task. Low BLEU scores—despite strong biological performance—support that results are not due to data leakage or memorization.
Task
Validation Metric
Spearman ρ
p -value
CTA
Ontology path distance
0.6212
<10−3
CG
% correct CellMarker genes
0.6669
<10−3
PP
DEG cosine similarity
0.6057
<10−3
CC
Expert preference alignment
0.8394
<10−3
SQA
Expert preference alignment
0.8199
<10−3
Appendix
Table 10: Reliability validation of the knowledge-augmented evaluator across SC-ARENA tasks. Metrics reflect correlation between evaluator scores and external validity measures (Spearman ρ and p -value). For open-ended tasks, we report expert agreement rate and correlation with evaluator rankings.
Figure 3: Radar-plot comparison of representative general-purpose models across the five SC-ARENA tasks: cell type annotation, perturbation prediction, cell generation, cell captioning, and scientific QA. The visualization highlights the uneven distribution of gains: while models such as Kimi-K2 and DeepSeek-R1 excel in captioning and generation, Qwen3-32B performs comparatively better in perturbation prediction. The radar plot provides a task-level perspective that complements aggregate scores and illustrates persistent challenges in mechanistic reasoning.
Figure 4: Distribution of ontology path length to root for predicted cell types across models. The x-axis shows binned intervals of the average path length to the ontology root, using left-closed, right-open notation [a,b), and the y-axis indicates the number of predicted cell types falling into each interval. Shorter path lengths indicate closer alignment with the ontology hierarchy and thus more specific predictions.
Robustness Test
Spearman ρ
Kendall’s τ
Cosine Similarity
Cross-model evaluator consistency
0.6701
0.5719
0.8873
Insensitivity to response length
0.0730
0.0578
–
Robustness to evaluation stochasticity (CTA task)
0.9070
0.8354
0.9468
Appendix
Table 11: Quantitative metrics for robustness validation of the knowledge-augmented evaluator.
Figure 5: Cell Type Annotation Answer Generation Prompt.
Single-cell studies require analysts to convert raw measurements into specific biological claims through multi-step workflows and integration of metadata, assay context, and auxiliary evidence. Existing AI-biology benchmarks largely measure broad knowledge, executable workflows, or local analysis steps. We introduce scBench-Long, a benchmark for long-horizon single-cell biology in which agents must recover scientific conclusions from raw or near-raw data without prescribed methods. The benchmark contains 21 evaluations spanning melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human--monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Tasks cover paired scRNA/TCR sequencing, RNA and chromatin profiling, cross-species transcriptomics, combinatorial scRNA-seq, single-nucleus RNA-seq, immune repertoires, ortholog maps, ligand--receptor resources, and validation evidence. Candidate claims are reproduced, reviewed, and converted into controlled answer vocabularies with deterministic grading and trajectory rubrics. Across 1,068 completed trajectories, the strongest model--harness pair passes 16/63 runs (25.4%). scBench-Long evaluates whether agents can move beyond local analysis steps and make complex scientific claims that are supported by single-cell data.
Automated single-cell annotation is difficult when the most abundant genes are not the most discriminative ones, or when a target state is poorly covered by a fixed reference atlas. GPTCelltype-style one-shot prompting allows large language models (LLMs) to produce plausible labels from generic expression signals, while reference-based annotators can force unfamiliar states into the nearest known category. We propose MAT-Cell, a prompt-driven framework for batch-level single-cell annotation that separates evidence grounding from label decision. MAT-Cell first uses Reverse Verification Query (RVQ) to combine tissue context, observed differentially expressed genes, and LLM-elicited biological priors into structured candidate-specific premises. Verifier agents then convert these premises into explicit premise-to-claim reasoning trees, and bounded multi-round debate compares,challenges, and revises the resulting claims before consensus or final adjudication.The returned Syllogistic Derivation Tree (SDT) provides an auditable debate trace rather than a formal proof of the annotation. In open-candidate benchmarks across five datasets, a locally deployed Qwen3-30B model with MAT-Cell achieves 75.5% average accuracy, compared with 64.2% for the strongest evaluated CoT baseline and 51.9% for the strongest evaluated scPilot variant. In oracle-candidate bench-marks across three species,MAT-Cell remains competitive across backbones, and local inference substantially reduces monetary cost for batch annotation. Code is available at: https://anonymous.4open.science/r/MATCell-4067
Yehui Yang, Zelin Zang, Xienan Zheng +9
Westlake University, Hangzhou, China · Shenzhen University of Advanced Technology, Shenzhen, China · Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China +1
Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom.
Yiyang Gu, Junwei Yang, Junyu Luo +15
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5