Evaluating and improving domain-specific LLM responses requires high-quality reference data and rubrics, yet expert authoring is slow and expensive, and synthetic datasets may fail to capture practitioner preferences unless they are grounded in expert judgment. We present SME-C (Subject Matter Expert Critique), a human-AI workflow that captures expertise through lightweight critiques of candidate responses and complements it with LLM-driven analysis and refinement to iteratively curate response datasets and evaluation rubrics. We validate SME-C in the domain of financial business briefs, collecting 3,853 critiques from 411 finance professionals on 520 queries across two survey rounds. We evaluate two LLM-driven refinement strategies: one guided by the experts' written critiques and one that infers critique solely from their preference votes. In a blind validation survey with 100 additional professionals, we found that both strategies significantly improved quality over the experts' top-ranked baseline responses. The gains were concentrated among weaker original responses, and the two strategies excelled in different ways: critique-based refinement produced more domain-grounded improvements, whereas vote-based refinement was more effective at identifying and removing irrelevant content. The resulting rubrics and refined dataset support reproducible offline evaluation and benchmarking of LLMs, suggesting that structured expert critique is a viable alternative to expert authoring for building high-quality, domain-specific evaluation resources more quickly, at scale, and at lower cost.
Figures & tables
Figure 1: Overview of the SME-C methodology. Stage 1 elicits SME critiques of LLM-generated responses through a survey protocol, Stage 2 analyzes the feedback qualitatively and quantitatively, and Stage 3 produces rubrics and refined responses through LLM-based refinement. The refined responses can seed the next iteration.
Figure 2: Example survey page showing a query about corporate strategy with three LLM-generated responses. Participants indicated their preferred response on three dimensions and provided open-ended feedback on strengths and suggested improvements.
Query Categories
Business Segments
Compliance, Risk & Legal/Regulatory
Culture & People
Customer & Market Insights
Financial Performance & Financial Health
Geographic & Governmental Insights
Table 1: Categories from which the queries were drawn.
Iteration 1 (Pilot)
Iteration 1 (Full-scale)
Iteration 2 (Validation)
Purpose
Validate instrument; collect initial critiques
Scale critique collection
Evaluate Stage 3 refinements
Queries
101
419
50 (subset of Full-scale)
Participants
107
304
100
Evals per participant
10
10
5
Evals per query
μ≈ 10 (range 8–11)
μ≈ 7 (range 3–10)
μ≈ 10 (range 5–14)
Total evaluations
1,042
2,811
500
Table 2: Overview of the three survey studies. The Iteration 1 pilot and full-scale rounds collected SME critiques (Stage 1); Iteration 2 compared the original Iteration 1 responses with their Stage 3 refinements and can inform further refinement. All participants were finance and business professionals recruited through an expert panel platform.
Dimension
Model A
Model B
Model C
Overall
37.0%
27.1%
35.9%
Content
37.4%
30.6%
32.0%
Format
37.7%
24.0%
38.3%
Table 3: Distribution of SME preference votes by model across both Iteration 1 Stage 1 surveys. Votes of “All about the same” are excluded.
Feedback Theme
Mentions
%
What respondents liked:
Depth of information
391
38%
Clarity and brevity
305
30%
Summary and comparison
164
16%
Requested improvements:
Improve section headers
209
20%
Table 4: Frequency of feedback themes in SME critiques from the Iteration 1 pilot ( N=1,042 evaluations). “What respondents liked” captures qualities that participants valued; “Requested improvements” captures the changes they suggested.
Model A
Model B
Model C
Refinement
Critique
Vote
Critique
Vote
Critique
Vote
Tables
84%
72%
92%
84%
85%
79%
Summaries
81%
69%
80%
65%
74%
59%
Citations
56%
45%
60%
69%
74%
92%
Length
98%
100%
90%
100%
95%
98%
Best Practices
—
73%
—
68%
—
77%
Table 5: Types of changes made by Critique-based refinement (guided by SME critiques) and Vote-based refinement (guided by the SME-voted best responses), relative to the original responses. Critique-based refinement changed tables and summaries more often, whereas Vote-based refinement generally made more changes to citations, response length, and best-practice content.
Metric
Original
Critique
Vote
Mean quality (1–10)
6.87
7.33
7.29
Median quality
7.0
8.0
8.0
Std. dev.
2.09
1.84
1.88
Table 6: Quality ratings by condition in the validation survey ( N=500 evaluations from 100 participants). Both refined conditions significantly outperform the Original (Holm-corrected paired t -tests, pHolm≤0.0012 ) but do not differ from each other ( pHolm=0.70 ).
Figure 3: Distribution of SME quality ratings in the Original, Critique-based, and Vote-based conditions. Violin plots show the density of the 1–10 ratings, embedded box plots show the interquartile ranges, and diamonds mark the means.
Figure 4: Relationship between perceived response quality and the need for further refinement. (A) Critique-based responses received the largest share of “no further improvement needed” ratings. (B) Mean quality decreases as the severity of the required improvement increases ( ρ=−0.329 , p<0.001 ).
Figure 5: Mean quality uplift of Critique-based and Vote-based refinement relative to the Original responses, by query category. Both methods improve categories such as Technology & Integration and Business Segments , whereas declines appear in categories including Compliance, Risk & Legal/Regulatory and Product & Service Intelligence . Vote-based refinement shows stronger gains in trend- and context-oriented categories.
Query Category
Original
Critique
Vote
N
Business Segments
6.19
7.45
7.39
31
Compliance, Risk & Legal/Regulatory
7.73
7.22
7.32
37
Culture & People
6.93
6.56
7.44
41
Customer & Market Insights
7.62
7.15
7.62
13
Financial Performance & Financial Health
6.75
7.48
7.34
209
Geographic & Governmental Insights
6.57
7.86
7.12
42
Table 7: Mean quality ratings (1–10) by query category in the validation survey. N is the number of evaluations per category (each evaluation is one participant rating one query).
Figure 6: Iteration 1 survey page for Query 2216. SMEs compared three LLM-generated responses side by side; Response C (highlighted) was selected as best overall by all nine evaluators.
SME Role
Industry
Why Best (Content/Format)
Suggested Improvements
VP, IT
Financial Services
“Answer B uses approximation and would be considered less reliable. C is more complete.”
“Prefer the most current first. Include graphical visuals below the table.”
Manager, Business Development
Distribution & Logistics
“Relevant data, complete response, plus extra info.” “Combination of free text and a table is very clear.”
“I’d include data for 2025 or at least a projection.”
Table 8: Representative Iteration 1 critiques for Query 2216; all nine SMEs selected Response C. Critiques are lightly edited for brevity.
Figure 7: Iteration 2 blind validation page for Query 2216. The three versions (Original, Critique-refined, and Vote-refined) were assigned to randomized letter positions; the Critique-refined version (here, Response C) was selected as best by 7 of the 10 participants.
Job Title
Industry
Best Pick
Why Best
Suggested Improvements
Director of Finance
Education
Critique
“This speaks to exactly what drove margin changes over the 5-year period for this company rather than the industry.”
—
VP, Finance & Ops.
Technology
Vote
“The most detailed explanation of T-Mobile’s profit and net margin trends. I love the inclusion of the visual trend map.”
“Include further detail on outlook and profitability metrics going forward.”
Investment Assistant
Financial Services
Critique
“It provided everything needed and was a lot more compact and readable.”
—
Table 9: Representative Iteration 2 feedback for Query 2216. Participants rated the three blinded versions on a 1–10 scale and explained their choice of the best version.