As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficiency in complex, open-ended tasks characterizing genuine expert-level cognition. Existing frameworks suffer from narrow domain coverage, reliance on generalist tasks, or self-evaluation biases. To bridge this gap, we present XpertBench, a high-fidelity benchmark engineered to assess LLMs across authentic professional domains. XpertBench consists of 1,346 meticulously curated tasks across 80 categories, spanning finance, healthcare, legal services, education, and dual-track research (STEM and Humanities). These tasks are derived from over 1,000 submissions by domain experts--including researchers from elite institutions and practitioners with extensive clinical or industrial experience--ensuring superior ecological validity. Each task uses detailed rubrics with mostly 15-40 weighted checkpoints to assess professional rigor. To facilitate scalable yet human-aligned assessment, we introduce ShotJudge, a novel evaluation paradigm that employs LLM judges calibrated with expert few-shot exemplars to mitigate self-rewarding biases. Our empirical evaluation of state-of-the-art LLMs reveals a pronounced performance ceiling: even leading models achieve a peak success rate of only ~66%, with a mean score around 55%. Models also exhibit domain-specific divergence, showing non-overlapping strengths in quantitative reasoning versus linguistic synthesis.. These findings underscore a significant "expert-gap" in current AI systems and establish XpertBench as a critical instrument for navigating the transition from general-purpose assistants to specialized professional collaborators.
Figures & tables
Figure 1: Overview of the benchmark construction and evaluation pipeline, consisting of expert recruitment, task curation, rubric design, and the SHOTJUDGE evaluation framework.
Figure 2: Distribution of tasks across professional domains in XpertBench, highlighting the balance between STEM and Humanities fields.
Domain
# Tasks
# of Rubrics
Prompt Length
Primary Competency
Finance
244
17
533
Market Analysis
Law
215
18
413
Juridical Reasoning
Education
329
15
960
Pedagogy & Planning
EAS
275
19
699
Technical Mechanism
HSS
116
15
373
Cultural Interpretation
Computer Science
92
17
558
Logical Deduction
Table 1: Detailed Statistics of XpertBench across Professional Domains. EAS stands for Engineering and Applied Sciences; HSS stands for Humanities and Social Sciences. Healthcare refers to Healthcare & Medicine.
Dimension
Description
Instruction Following
Accurately identifying and understanding task instructions, requirements, and goals.
Constraint Satisfaction
Strictly adhering to explicit constraints and requirements stipulated in the prompt.
Accuracy & Factual Correctness
Precision and integrity of domain-specific concepts, theories, regulations, and factual data.
Information Processing
Ability to extract, synthesize, and integrate information from literature or given contexts.
Logical Coherence
Rigor of reasoning, completeness of argumentation, and sound analysis of causal relationships.
Domain Expertise
Application of core industry mindsets, critical thinking, analytical methods, tools, and theoretical frameworks.
Table 2: Comprehensive Evaluation Rubric Dimensions. This table outlines the specific dimensions and their corresponding detailed descriptions used to assess and score candidate model responses.
Figure 3: Results on XpertBench-Gold evaluation subset (N=245).
Model
Overall
Finance
Law
Education
EAS
HSS
Claude-Opus-4.6-thinking
66.20%
73.25%
65.54%
57.96%
49.58%
83.02%
GPT-5.4-high
64.78%
84.65%
64.79%
59.29%
42.84%
80.58%
Doubao-2.0-pro
64.51%
73.82%
65.06%
55.35%
44.88%
80.09%
GPT-5.2-high
61.03%
71.22%
56.88%
58.43%
46.13%
76.15%
GPT-5-high
58.70%
64.50%
54.70%
56.90%
48.20%
68.50%
Gemini-3.1-pro
55.05%
62.43%
53.88%
52.43%
37.30%
69.04%
Table 3: Model Performance Breakdown on the XpertBench-Gold subset (N=245) across Different Domains. The highest score in each category is highlighted in bold. EAS stands for Engineering and Applied Sciences; HSS stands for Humanities and Social Sciences.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Criterion
Points
Importance
Tag
Based on FY2023 data, calculate Lockheed Martin’s book-to-bill ratio as approximately 1.16. The calculation must use correct net order amounts and total revenue. A ± 2% margin of error is allowed.
10
Essential
Authenticity
Based on FY2023 data, calculate Northrop Grumman’s book-to-bill ratio as approximately 1.14. The calculation must use correct net order amounts and total revenue. A ± 2% margin of error is allowed.
10
Essential
Authenticity
Accurately explain the meaning of a book-to-bill ratio (Book-to-Bill) above 1.0, i.e., new orders exceed current period revenue, indicating strong future revenue visibility.
10
Essential
Depth
Accurately identify LMT’s business segment with the highest operating margin in 2023 as “Missiles and Fire Control.”
10
Essential
Authenticity
Accurately state that the “Missiles and Fire Control” segment’s operating margin is approximately 13.7%. A ± 0.3% margin of error is allowed.
10
Essential
Authenticity
Accurately identify Northrop Grumman’s business segment with the highest operating margin in 2023 as “Mission Systems.”
10
Essential
Authenticity
Appendix
Table A.1: Scoring rubric for the Finance example task.
Criterion
Points
Importance
Tag
Accurately identify the legal relationship between Guangxi Company and China Construction Bank as accounts receivable pledge financing, rather than a factoring contract relationship.
7
Important
Requirement Identification
Ability to clarify the core reasons for determining a loan contract relationship, i.e., the financing behavior in this case lacks the core element of “accounts receivable transfer” required for a factoring contract.
7
Important
Logicality
Accurately analyze that lack of factoring qualifications does not necessarily render the Factoring Financing Agreement invalid, and reach a conclusion favoring contract validity.
7
Important
Professionalism
Ability to cite key legal principles or provisions to argue why lack of factoring qualifications does not constitute an “effectiveness-mandatory provision” that would invalidate the contract.
7
Important
Professionalism
Accurately point out that in recourse factoring, the factor (financing guarantee company) has no sequential restrictions when asserting rights against the creditor (Guangxi Company) and the debtor (Yunnan Company).
7
Important
Accuracy
Ability to analyze the legal consequences and liability allocation for all parties if the Factoring Financing Agreement is deemed invalid.
7
Important
Logicality
Appendix
Table A.2: Scoring rubric for the Law example task.
Criterion
Points
Importance
Tag
Teaching objectives adopt “three-dimensional objectives” with point-by-point elaboration, and each dimension must list at least one observable, measurable specific behavioral objective.
7
Important
Standardization
Each dimension of teaching objectives requires layered design, with clear evaluation criteria set for three categories of students: better, moderate, and more severe.
7
Important
Completeness
Each dimension of teaching objectives must use measurable action verbs and provide specific conditions or standards (e.g., “able to read correctly within 3 attempts”).
7
Important
Standardization
Teaching focus must clearly state “able to write time format correctly (HH:MM)” and provide one example.
7
Important
Instruction Following
Difficulty explanation should simultaneously provide the main methods to overcome difficulties and corresponding teaching aids.
10
Essential
Completeness
All listed teaching methods must appear in at least two specific application scenarios in the subsequent teaching process.
7
Important
Completeness
Appendix
Table A.3: Scoring rubric for the Education example task.
Figure A.1 : The recognition sequence and cleavage sites (indicated by triangles) of the restriction enzymes Mbo I and Sau 3AI. Both enzymes recognize the same 5′ -GATC- 3′ nucleotide sequence.
Figure A.2 : Agarose gel electrophoresis results for plasmid verification. Top (Figure 1): Digestion profiles of plasmids extracted from E. coli DH5 α . Lanes 1–20 are samples treated with Mbo I; lanes 21–22 (CE) are undigested control plasmids. Bottom (Figure 2): Digestion profiles of the same plasmids after being transferred to and extracted from E. coli ET12567. The left panel shows digestion with Mbo I, and the right panel shows digestion with Sau 3AI. ’M’ denotes the DNA molecular weight marker (ladder).
Criterion
Points
Importance
Tag
Analyze the reason for unsuccessful digestion: Dam methylation modification in DH5 α blocks restriction enzyme digestion.
10
Essential
Information Processing
Determine the state of fragments in the map: bands represent circular plasmids whose migration rate is affected by conformation (supercoiled, open circular, etc.); they cannot reflect actual size and cannot be used to determine empty vectors.
10
Essential
Professionalism
Observe the restriction map and determine that plasmids have been successfully digested into linear fragments. The sum of bands can reflect the total length of “vector + foreign fragment,” which is the key basis for determining empty vectors.
10
Essential
Professionalism
Analyze the reason for successful digestion: ET12567 is a methylation-deficient strain (dam - dcm - ), and digestion successfully produces linear fragments.
7
Important
Professionalism
Explain the difference in migration rate between circular plasmids (especially supercoiled) in electrophoresis and linear DNA used as markers (supercoiled migrates faster than linear DNA of the same size).
7
Important
Professionalism
Point out that bands near 10 kb in Figure 1 only represent circular plasmid conformations, unrelated to the actual size of the plasmid (vector + foreign fragment), and cannot directly determine whether it is empty based on position.
10
Essential
Information Processing
Appendix
Table A.4: Scoring rubric for the STEM example task (Part 1).
Criterion
Points
Importance
Tag
Content accuracy and depth, restoration of Confucius’s philosophical thought: Accurately and three-dimensionally present Confucius’s view on life and death, integrating the pragmatism of “not yet knowing life, how can one know death,” the value pursuit of “hearing the Way in the morning, one can die content in the evening,” and the tension of “fearing Heaven’s mandate while doing one’s best,” with “benevolence” and “ritual” as the fundamental starting point of all his thoughts and actions.
10.0
Essential
Professionalism
Qualification review (threshold requirement), comprehensive compliance review: This checks whether the work fully meets the basic requirements in format, language, content setting, and factual accuracy. The work must simultaneously satisfy all of the following conditions: 1. Format completeness. 2. Language standardization (prohibition of semi-classical/modern Chinese mixed with classical Chinese, etc.). 3. Premise compliance (Confucius and Socrates meeting in their final years and the “ignoring temporal/spatial/language” setting). 4. Historical accuracy, no tampering with key historical facts. 5. Dialogue balance, presented as bidirectional, back-and-forth dialogue, rather than continuous one-way responses or lecture mode.
10.0
Essential
Format Structure
Content accuracy and depth, restoration of Socrates’s philosophical thought: Accurately present his core idea that “the unexamined life is not worth living,” treating “death” as the ultimate test of “care of the soul,” and embodying his unique method of argumentation (maieutics).
10.0
Essential
Professionalism
Content accuracy and depth, use of historical background and events: Ability to use the macro situation of both figures being at “the turning point from prosperity to decline, seeking the Way but not meeting it” as the narrative tone, accurately employing historical facts such as “Western hunt capturing the qilin” and “Athenian trial,” making them internal drivers of the philosophical debate rather than merely external plot devices. All historical details (characters, time) are accurate.
10.0
Essential
Accuracy
Core task completion, effectiveness and depth of intellectual confrontation: Truly achieves intellectual confrontation and collision, with both sides’ arguments able to question each other and progress layer by layer, rather than “talking past each other.” The debate can go from the surface of views on life and death to the fundamental differences behind views on Heaven’s mandate, truth, soul, and social responsibility.
10.0
Essential
Logicality
Core task completion, logical progression of debate: The debate process is logically clear, with questions interlocking, naturally transitioning from one argument to the next, ultimately leading to a meaningful dramatic climax or philosophical reflection. If characters undergo conceptual changes, clearly present the continuous trajectory “from evidence to psychological inference to behavioral/linguistic representation.”
10.0
Essential
Logicality
Appendix
Table A.5: Scoring rubric for the HSS example task (Part 2).
As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce $OneMillion-Bench ($OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, $OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.
Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to HealthBench and DR.BENCH, covering medical and 10-domain professional evaluation settings. Code and data are available at https://github.com/smiling-world/JADE.
Lanbo Lin, Jiayao Liu, Tianyuan Yang +5
Alibaba International Digital Commerce Group · Zhejiang University · Peking University
Reliable evaluation of large language models (LLMs) is critical as their deployment rapidly expands, particularly in high-stakes domains such as business and finance. The LLM-as-a-Judge framework, which uses prompted LLMs to evaluate response quality, is appealing due to its scalability, low cost, and strong correlations with human stylistic preferences. However, it remains unclear how accurately these methods can assess response quality in domains where correctness matters more than style. To address this gap, we introduce the Business and Finance Fundamentals Benchmark (BFF-Bench), a dataset of 160 challenging questions and long-form responses authored by financial professionals. These experts subsequently evaluated the correctness of 1,200 responses generated by a diverse set of LLMs on both BFF-Bench and a challenging subset of MT-Bench. With this expert-annotated dataset of judgments (VERDICTS), we analyze the agreement between a suite of automated grading methods and human experts. While we observe that LLM Judges are more reliable than other grading methods, our findings reveal a clear pattern in LLM Judge performance: when not provided with a correct reference, judges show high agreement with human experts only on questions the judges were able to correctly answer themselves. We demonstrate that providing the judges with expert-written references largely mitigates this issue, highlighting the limits of using LLM-as-a-Judge without any form of human verification.
Michael Krumdick, Charles Lovering, Varshini Reddy +2