cs.AIMar 27, 2026
SaveXpertbench: Expert Level Tasks with Rubrics-Based Evaluation
Organizations: ByteDance Seed
Abstract
As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.
Figures & tables
Figure 1: Overview of the benchmark construction and evaluation pipeline, consisting of expert recruitment, task curation, rubric design, and the SHOTJUDGE evaluation framework.
Figure 2: Distribution of tasks across professional domains in XpertBench, highlighting the balance between STEM and Humanities fields.
| Domain | # Tasks | # of Rubrics | Prompt Length | Primary Competency |
| Finance | 244 | 17 | 533 | Market Analysis |
| Law | 215 | 18 | 413 | Juridical Reasoning |
| Education | 329 | 15 | 960 | Pedagogy & Planning |
| EAS | 275 | 19 | 699 | Technical Mechanism |
| HSS | 116 | 15 | 373 | Cultural Interpretation |
| Computer Science | 92 | 17 | 558 | Logical Deduction |
Table 1: Detailed Statistics of XpertBench across Professional Domains. EAS stands for Engineering and Applied Sciences; HSS stands for Humanities and Social Sciences. Healthcare refers to Healthcare & Medicine.
| Dimension | Description |
| Instruction Following | Accurately identifying and understanding task instructions, requirements, and goals. |
| Constraint Satisfaction | Strictly adhering to explicit constraints and requirements stipulated in the prompt. |
| Accuracy & Factual Correctness | Precision and integrity of domain-specific concepts, theories, regulations, and factual data. |
| Information Processing | Ability to extract, synthesize, and integrate information from literature or given contexts. |
| Logical Coherence | Rigor of reasoning, completeness of argumentation, and sound analysis of causal relationships. |
| Domain Expertise | Application of core industry mindsets, critical thinking, analytical methods, tools, and theoretical frameworks. |
Table 2: Comprehensive Evaluation Rubric Dimensions. This table outlines the specific dimensions and their corresponding detailed descriptions used to assess and score candidate model responses.
Figure 3: Results on XpertBench-Gold evaluation subset (N=245).
| Model | Overall | Finance | Law | Education | EAS | HSS |
| Claude-Opus-4.6-thinking | 66.20% | 73.25% | 65.54% | 57.96% | 49.58% | 83.02% |
| GPT-5.4-high | 64.78% | 84.65% | 64.79% | 59.29% | 42.84% | 80.58% |
| Doubao-2.0-pro | 64.51% | 73.82% | 65.06% | 55.35% | 44.88% | 80.09% |
| GPT-5.2-high | 61.03% | 71.22% | 56.88% | 58.43% | 46.13% | 76.15% |
| GPT-5-high | 58.70% | 64.50% | 54.70% | 56.90% | 48.20% | 68.50% |
| Gemini-3.1-pro | 55.05% | 62.43% | 53.88% | 52.43% | 37.30% | 69.04% |
Table 3: Model Performance Breakdown on the XpertBench-Gold subset (N=245) across Different Domains. The highest score in each category is highlighted in bold. EAS stands for Engineering and Applied Sciences; HSS stands for Humanities and Social Sciences.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Criterion | Points | Importance | Tag |
|---|---|---|---|
| Based on FY2023 data, calculate Lockheed Martin’s book-to-bill ratio as approximately 1.16. The calculation must use correct net order amounts and total revenue. A 2% margin of error is allowed. | 10 | Essential | Authenticity |
| Based on FY2023 data, calculate Northrop Grumman’s book-to-bill ratio as approximately 1.14. The calculation must use correct net order amounts and total revenue. A 2% margin of error is allowed. | 10 | Essential | Authenticity |
| Accurately explain the meaning of a book-to-bill ratio (Book-to-Bill) above 1.0, i.e., new orders exceed current period revenue, indicating strong future revenue visibility. | 10 | Essential | Depth |
| Accurately identify LMT’s business segment with the highest operating margin in 2023 as “Missiles and Fire Control.” | 10 | Essential | Authenticity |
| Accurately state that the “Missiles and Fire Control” segment’s operating margin is approximately 13.7%. A 0.3% margin of error is allowed. | 10 | Essential | Authenticity |
| Accurately identify Northrop Grumman’s business segment with the highest operating margin in 2023 as “Mission Systems.” | 10 | Essential | Authenticity |
Table A.1: Scoring rubric for the Finance example task.
| Criterion | Points | Importance | Tag |
|---|---|---|---|
| Accurately identify the legal relationship between Guangxi Company and China Construction Bank as accounts receivable pledge financing, rather than a factoring contract relationship. | 7 | Important | Requirement Identification |
| Ability to clarify the core reasons for determining a loan contract relationship, i.e., the financing behavior in this case lacks the core element of “accounts receivable transfer” required for a factoring contract. | 7 | Important | Logicality |
| Accurately analyze that lack of factoring qualifications does not necessarily render the Factoring Financing Agreement invalid, and reach a conclusion favoring contract validity. | 7 | Important | Professionalism |
| Ability to cite key legal principles or provisions to argue why lack of factoring qualifications does not constitute an “effectiveness-mandatory provision” that would invalidate the contract. | 7 | Important | Professionalism |
| Accurately point out that in recourse factoring, the factor (financing guarantee company) has no sequential restrictions when asserting rights against the creditor (Guangxi Company) and the debtor (Yunnan Company). | 7 | Important | Accuracy |
| Ability to analyze the legal consequences and liability allocation for all parties if the Factoring Financing Agreement is deemed invalid. | 7 | Important | Logicality |
Table A.2: Scoring rubric for the Law example task.
| Criterion | Points | Importance | Tag |
|---|---|---|---|
| Teaching objectives adopt “three-dimensional objectives” with point-by-point elaboration, and each dimension must list at least one observable, measurable specific behavioral objective. | 7 | Important | Standardization |
| Each dimension of teaching objectives requires layered design, with clear evaluation criteria set for three categories of students: better, moderate, and more severe. | 7 | Important | Completeness |
| Each dimension of teaching objectives must use measurable action verbs and provide specific conditions or standards (e.g., “able to read correctly within 3 attempts”). | 7 | Important | Standardization |
| Teaching focus must clearly state “able to write time format correctly (HH:MM)” and provide one example. | 7 | Important | Instruction Following |
| Difficulty explanation should simultaneously provide the main methods to overcome difficulties and corresponding teaching aids. | 10 | Essential | Completeness |
| All listed teaching methods must appear in at least two specific application scenarios in the subsequent teaching process. | 7 | Important | Completeness |
Table A.3: Scoring rubric for the Education example task.
Figure A.1 : The recognition sequence and cleavage sites (indicated by triangles) of the restriction enzymes Mbo I and Sau 3AI. Both enzymes recognize the same -GATC- nucleotide sequence.
Figure A.2 : Agarose gel electrophoresis results for plasmid verification. Top (Figure 1): Digestion profiles of plasmids extracted from E. coli DH5 . Lanes 1–20 are samples treated with Mbo I; lanes 21–22 (CE) are undigested control plasmids. Bottom (Figure 2): Digestion profiles of the same plasmids after being transferred to and extracted from E. coli ET12567. The left panel shows digestion with Mbo I, and the right panel shows digestion with Sau 3AI. ’M’ denotes the DNA molecular weight marker (ladder).
| Criterion | Points | Importance | Tag |
|---|---|---|---|
| Analyze the reason for unsuccessful digestion: Dam methylation modification in DH5 blocks restriction enzyme digestion. | 10 | Essential | Information Processing |
| Determine the state of fragments in the map: bands represent circular plasmids whose migration rate is affected by conformation (supercoiled, open circular, etc.); they cannot reflect actual size and cannot be used to determine empty vectors. | 10 | Essential | Professionalism |
| Observe the restriction map and determine that plasmids have been successfully digested into linear fragments. The sum of bands can reflect the total length of “vector foreign fragment,” which is the key basis for determining empty vectors. | 10 | Essential | Professionalism |
| Analyze the reason for successful digestion: ET12567 is a methylation-deficient strain (dam - dcm - ), and digestion successfully produces linear fragments. | 7 | Important | Professionalism |
| Explain the difference in migration rate between circular plasmids (especially supercoiled) in electrophoresis and linear DNA used as markers (supercoiled migrates faster than linear DNA of the same size). | 7 | Important | Professionalism |
| Point out that bands near 10 kb in Figure 1 only represent circular plasmid conformations, unrelated to the actual size of the plasmid (vector foreign fragment), and cannot directly determine whether it is empty based on position. | 10 | Essential | Information Processing |
Table A.4: Scoring rubric for the STEM example task (Part 1).
| Criterion | Points | Importance | Tag |
|---|---|---|---|
| Content accuracy and depth, restoration of Confucius’s philosophical thought: Accurately and three-dimensionally present Confucius’s view on life and death, integrating the pragmatism of “not yet knowing life, how can one know death,” the value pursuit of “hearing the Way in the morning, one can die content in the evening,” and the tension of “fearing Heaven’s mandate while doing one’s best,” with “benevolence” and “ritual” as the fundamental starting point of all his thoughts and actions. | 10.0 | Essential | Professionalism |
| Qualification review (threshold requirement), comprehensive compliance review: This checks whether the work fully meets the basic requirements in format, language, content setting, and factual accuracy. The work must simultaneously satisfy all of the following conditions: 1. Format completeness. 2. Language standardization (prohibition of semi-classical/modern Chinese mixed with classical Chinese, etc.). 3. Premise compliance (Confucius and Socrates meeting in their final years and the “ignoring temporal/spatial/language” setting). 4. Historical accuracy, no tampering with key historical facts. 5. Dialogue balance, presented as bidirectional, back-and-forth dialogue, rather than continuous one-way responses or lecture mode. | 10.0 | Essential | Format Structure |
| Content accuracy and depth, restoration of Socrates’s philosophical thought: Accurately present his core idea that “the unexamined life is not worth living,” treating “death” as the ultimate test of “care of the soul,” and embodying his unique method of argumentation (maieutics). | 10.0 | Essential | Professionalism |
| Content accuracy and depth, use of historical background and events: Ability to use the macro situation of both figures being at “the turning point from prosperity to decline, seeking the Way but not meeting it” as the narrative tone, accurately employing historical facts such as “Western hunt capturing the qilin” and “Athenian trial,” making them internal drivers of the philosophical debate rather than merely external plot devices. All historical details (characters, time) are accurate. | 10.0 | Essential | Accuracy |
| Core task completion, effectiveness and depth of intellectual confrontation: Truly achieves intellectual confrontation and collision, with both sides’ arguments able to question each other and progress layer by layer, rather than “talking past each other.” The debate can go from the surface of views on life and death to the fundamental differences behind views on Heaven’s mandate, truth, soul, and social responsibility. | 10.0 | Essential | Logicality |
| Core task completion, logical progression of debate: The debate process is logically clear, with questions interlocking, naturally transitioning from one argument to the next, ultimately leading to a meaningful dramatic climax or philosophical reflection. If characters undergo conceptual changes, clearly present the continuous trajectory “from evidence to psychological inference to behavioral/linguistic representation.” | 10.0 | Essential | Logicality |
Table A.5: Scoring rubric for the HSS example task (Part 2).