EHR-Bench: A Chinese Multi-Task Benchmark for Recruitment AI

Ruijia Yang; Bing Xu; Zulong Chen; Zhipeng Xu; Jia Xu; Yipeng Yu; Ying Zhou; Zeyi Wen

cs.CLOct 12, 2026

EHR-Bench: A Chinese Multi-Task Benchmark for Recruitment AI

Authors: Ruijia Yang, Bing Xu, Zulong Chen, Zhipeng Xu, Jia Xu, Yipeng Yu, Ying Zhou, Zeyi Wen

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Alibaba Group · Guangzhou University · University of Electronic Science and Technology of China

Abstract

Recruitment applications span diverse operations, from drafting job descriptions and interpreting queries to organizing profiles, assessing fit, and recommending candidates and jobs. Systematic LLM evaluation therefore requires task coverage that reflects this range of business needs. We introduce EHR-Bench, an open Chinese benchmark comprising twelve tasks across six capability families and five recruitment scenarios. The tasks cover generation, extraction, classification, retrieval, semantic similarity, and reciprocal ranking within one evaluation framework. An internal recruitment dataset guides task definitions, the occupation taxonomy and sampling targets, output requirements, and scoring protocols. Following these specifications, we construct 8,336 task instances from public materials and controlled augmentation, with source-aware expert review and documented provenance. We evaluate twelve LLMs through task-level, capability-level, and aggregate comparisons. Closely ranked models exhibit distinct task strengths, and performance varies across input sources, reference evidence, and candidate pools. Five shared models retain the same observed Overall ordering on a private operational reference, with variation in capability and task correspondence. EHR-Bench provides a reproducible foundation for systematic model assessment, task-specific model selection, and further research on LLMs for recruitment. The project repository is available at https://github.com/EhrBench/EHR-Bench.

Figures & tables

Figure 1: Overview of EHR-Bench : twelve tasks, six capability families, and five recruitment scenarios. Examples are illustrative; English text is translated. Radars show five leading model series, with scores scaled from center to rim over the range labeled on each axis.

Resource	Language / output	Candidate conditions / scope
ResumeBench ( Ling et al., 2025 )	Five languages; PDF to JSON	Schema-guided resume parsing
PJB ( Wang et al., 2026 )	Chinese; resume retrieval	297 queries; 197,674-resume collection
WorkRB ( De Lange et al., 2026 )	Multilingual; ranking and classification	Ontology- and corpus-based candidate spaces
EHR-Bench (ours)	Chinese; text, spans, paths, relations, orderings	12 operations; 8–50-candidate retrieval and 5–10-candidate ranking pools

Table 1: Evaluation interfaces and candidate conditions. WorkRB coverage includes the ranking and classification interfaces in its accompanying toolkit ( TechWolf Research Team, 2026 ) .

Family	Task	Input $\rightarrow$ output	$n$	Main metric
Gen	Job description generation (JDG)	Job constraints $\rightarrow$ JD	941	$.6R+.4J$
	Person–job fit analysis (PJFA)	Resume + job $\rightarrow$ verdict, analysis	872	$.5V+.3R+.2J$
Ext	Query parsing (QP)	Query $\rightarrow$ typed spans	714	Strict span-F1
Cls	Job classification (JC)	Job text $\rightarrow$ category path	1,064	PathScore
	Talent classification (TC)	Resume $\rightarrow$ category path	161	PathScore
	Person–job fit (PJF)	Resume + job $\rightarrow$ binary fit	1,386	Balanced accuracy

Table 2: Task definitions and abbreviations used throughout the paper.

n

counts records (

\dagger

: query groups).

R

J

V

: ROUGE-L, rubric score, verdict accuracy; R@

k

: Recall@

k

Figure 2: Source composition by task. Colors indicate five categories of record provenance and construction; counts treat each retrieval or ranking group as one evaluation instance.

System	Overall	Gen	Ext	Cls	Ret	STS	Rank
Claude-opus-5	74.53	55.03	92.02	71.20	63.37	80.10	85.45
Kimi-K3	74.28	55.16	91.66	71.23	61.84	81.38	84.42
GPT-5.6-sol	74.10	54.56	91.68	72.14	60.84	80.05	85.32
Gemini-3.5-flash	74.01	53.00	90.31	72.10	60.54	82.69	85.41
Claude-opus-4.8	73.81	54.92	91.36	71.51	60.44	79.84	84.81
GLM-5.3	73.54	57.30	91.57	71.41	59.95	81.92	79.11

Table 3: Overall and capability scores. Bold / underline indicate the best / second-best distinct values, including ties. Overall averages the six capabilities.

Figure 3: Twelve-task scores in Overall order. Red/blue indicate scores below/above each task’s median, clipped at $\pm 15$ points; colors show within-task relative position, not cross-task difficulty.

Task	Source	$n$	Uniform	TF-IDF	LLM
TR	Public interactions	104	16.68	37.90	45.55
TR	Derived pools	80	53.34	72.22	86.53
J2T	Derived	110	43.10	56.97	74.16
J2T	Synthetic	6	44.75	13.63	76.03
T2J	Synthetic	361	36.86	89.05	97.19

Table 4: Candidate-task baselines by source (0–100). Uniform: expected score over permutations. LLM: the best whole-task system, scored within each stratum. Synthetic J2T has six groups.

Figure 4: Performance across evaluation conditions (0–100). (a) PJF: behavior-derived (

n=900

) and constructed (

486

); dashed line: chance at 50. (b) PJFA: interaction-derived (

500

) and synthetic (

372

). (c) SimT recall and negative rejection (

811

); numbers identify the systems listed at left.

Figure 5: Correspondence on five shared models. (a) Overall and capability Spearman correlations, with same-direction counts out of ten pairs. (b) Direction counts for each task at displayed precision; ties remain in the denominator of ten model pairs per task.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Task	External positive evidence	External negatives / construction	Internal counterpart
PJF	Inherited satisfaction labels in 900 public pairs; expert judgments in 486 added pairs	Applied without satisfaction (215); browsed without application (207); expert-reviewed new pairings	Resume assessment pass / fail
TR	Application in public interaction pools; constructed relevance in derived pools	Other candidates within the supplied pool	Talent selected for assessment after JD-search exposure; sampled negatives
SimT	Shared-job satisfaction seeds; same second-level occupation seeds	Same-category or same-first-level negatives; cross-category random pairs; review sets final labels	Both candidates pass the final interview for the same job
SimJ	Shared-candidate satisfaction seeds; occupation-based pairs	Same-category and cross-category pairs; review sets final labels	Same candidate passes final interviews for both jobs
Dedup	Masked, lossy, or multiple snapshots of one identity	Distinct-person neighbors matched on occupation, education, or seniority; reviewed labels	Expert-labeled same-person / different-person pairs
J2T	Satisfaction-derived relevance; small constructed supplement	Remaining candidates in each single-positive pool	Recommended talent accepted by the user

Appendix

Table 5: Operational label evidence for relation and candidate tasks. Final labels follow task-specific review of the initial construction.

Figure 6: Reference-label review SOP. R1 reviews all samples; R2 handles cases requiring further review; R3 adjudicates disagreements.

Task	Double-annotated records	Cohen’s $\kappa$
JC	98	0.89
TC	20	0.83
PJF	125	0.86
SimT	74	0.80
SimJ	78	0.82
Dedup	79	0.92

Appendix

Table 6: Independent annotation agreement before adjudication.

Operational evidence	Tasks	External design and construction guidance
Vacancy drafting: JD outputs adopted by platform users.	JDG	Required job fields, the JD-generation operation, and a shared quality rubric.
Candidate assessment: screening pass/reject decisions and HR-written interview assessments.	PJF, PJFA	Distinct fit-decision and verdict-plus-analysis outputs; shared analysis rubric.
Search understanding: recruitment queries with annotated entity spans.	QP	Typed entities, span-based extraction, and the query-output schema.
Profile organization: operationally maintained occupation labels for jobs and talents.	JC, TC	A shared hierarchical occupation space, category-path outputs, and occupation-based sampling targets.
Talent discovery: candidates scheduled for assessment after JD-search exposure.	TR	Job anchors, eligible candidate pools, and relevance-based retrieval evaluation.
Related-profile discovery: talents passing final interviews for the same job, or jobs for which the same talent passes.	SimT, SimJ	Recruitment-domain similarity criteria; shared interactions seed pairs for review, alongside random and occupation-related constructions.

Appendix

Table 7: Business operations, corresponding tasks, and external construction guidance.

Source tag	Records	Percent
public_dataset	695	8.3
public_job_descriptions	1,505	18.1
public_data_derived	3,732	44.8
controlled_synthetic	1,690	20.3
synthetic	714	8.6

Appendix

Table 8: Record-level source counts, including query groups. Percentages use 8,336 records and may not sum to 100 through rounding.

Resource / material	Construction use	Label evidence
Tianchi–Zhaopin competition data	Recruitment text and interaction-derived pairs or pools	Observed interaction fields for the original relation; task rules or reviewed judgments for constructed relations
iic/resume_ner	Resume text for talent classification and identity examples	Occupation labeling, same-person transformations, and negative pairs
Avera	Small Chinese adaptation used in reciprocal-ranking augmentation	Task-specific candidate relevance
Greenhouse company job boards	Cleaned JD references and occupation classification	Reference text or reviewed category labels
Controlled synthetic materials	Persona-based resumes and other task-specific constructions	Expert labels for persona-generated resumes; task-specific rules and review elsewhere

Appendix

Table 9: Source resources, construction uses, and label evidence.

Company	Board token	Company	Board token
Stripe	stripe	Planet Labs	planetlabs
Anthropic	anthropic	Esri	esri
Coinbase	coinbase	Gotion	gotion
Cloudflare	cloudflare	Flexport	flexport
Roblox	roblox	project44	project44
Taboola	taboola	ShipMonk	shipmonk

Appendix

Table 10: Company boards used for public JD collection; tokens link to JSON endpoints.

Task	Illustrative input	Target form and task meaning
JDG	Data engineer; SQL; maintain data pipelines.	A reference job description stating the role’s duties and requirements.
PJFA	Resume mentions SQL pipelines; job requires SQL and pipeline maintenance.	verdict: pass ; analysis_text : explanation of the stated experience and requirements.
QP	“Find SQL engineers in City A.”	skill_name: [SQL] , jobname: [engineers] , city: [City A] ; surface strings are mapped back to exact query spans.
JC	A job description focused on data-pipeline development.	cate_labels : an ordered occupation path selected from the supplied taxonomy.
TC	A resume describing data-pipeline development experience.	cate_labels : the talent’s occupation path in that taxonomy, rather than a label for a job vacancy.
PJF	A supplied resume–job pair.	satisfied: 1 or 0 ; the reference relation follows the pair’s behavioral or constructed label evidence.

Appendix

Table 11: Illustrative task inputs and outputs; field names follow evaluation prompts.

Setting	Implementation
Prompt / output	Chinese task instructions; plain-text JD generation, task-specific JSON otherwise. Retrieval and ranking request all candidate IDs.
Temperature (runs)	0.0; Kimi-K3 omits this parameter and uses the endpoint default.
Reasoning (runs)	Thinking disabled where configurable; GLM-5.3 uses low reasoning effort.
Output budget (runner)	Default 4,096 tokens outside ranking; ranking uses $\max(16{,}384,\,60N+8{,}192)$ for $N$ candidates.
Input truncation	Occupation: 3,500 characters; PJF/PJFA: 3,000 per text; retrieval candidate experience: default 300; ranking candidate profile: 400.

Appendix

Table 12: Decoding settings and input/output budgets.

Task	Dimensions	Score formation
JDG	Structure; alignment with job requirements; professionalism; actionable requirements; concision; consistency; attractiveness	Each dimension 0–3; weights 3, 3, 2, 2, 1, 2, 2; normalized to 0–100.
PJFA	Reasonableness of strengths/weaknesses; supporting evidence; clarity of structure; professionalism	Four dimensions of 0–25; total 0–100.

Appendix

Table 13: Rubric dimensions and their scoring scales in the rubric judge.

	Gen		Ext	Cls			Ret	STS			Rank
System	JDG	PJFA	QP	JC	TC	PJF	TR	SimT	SimJ	Dedup	J2T	T2J
Claude-opus-5	45.81	64.25	92.02	79.35	65.53	68.71	63.37	60.80	81.11	98.39	73.72	97.19
Kimi-K3	44.63	65.68	91.66	78.48	65.53	69.68	61.84	68.04	80.70	95.40	72.78	96.06
GPT-5.6-sol	43.84	65.28	91.68	80.26	66.15	70.01	60.84	61.23	81.44	97.47	73.49	97.16
Gemini-3.5-flash	44.62	61.39	90.31	79.87	65.53	70.91	60.54	70.71	80.93	96.44	74.26	96.56
Claude-opus-4.8	44.03	65.82	91.36	79.51	66.46	68.56	60.44	63.86	81.32	94.35	73.66	95.96

Appendix

Table 14: Complete external task scores. Bold / underline mark the best / second-best distinct displayed values, including ties. Abbreviations follow Table 2 .

Task	Scored / total	Main score	Refusals as failures
JC	1063/1064	76.36	76.28
SimJ	855/856	79.47	79.41

Appendix

Table 15: MiniMax-M3 scores with refusals omitted or counted as failures.

Figure 7: Twelve-system diagnostics in Overall order. (a) PJFA components (0–100; $n=872$ ). (b) JC prefix hit rates ( $n=1{,}064$ ; MiniMax-M3: $1{,}063$ ). (c) T2J nDCG@5 on all 361 groups versus valid groups, with valid/total counts.

	Job description generation			Person–job fit analysis
System	ROUGE-L	Rubric	Score	Verdict	ROUGE-L	Rubric	Score
Claude-opus-5	17.01	89.01	45.81	80.96	17.24	92.96	64.25
Kimi-K3	20.01	81.54	44.63	83.60	20.03	89.34	65.68
GPT-5.6-sol	19.70	80.04	43.84	80.05	22.25	92.93	65.28
Gemini-3.5-flash	16.54	86.74	44.62	75.46	19.26	89.43	61.39
Claude-opus-4.8	20.43	79.43	44.03	84.40	20.67	87.09	65.82

Appendix

Table 16: Generation components (0–100). JDG weights ROUGE-L/rubric by 0.6/0.4; PJFA weights verdict/ROUGE-L/rubric by 0.5/0.3/0.2.

	Job classification			Talent classification
System	L1	L1–2	L1–3	L1	L1–2
Claude-opus-5	87.4 (1064)	71.5 (1064)	87.5 (32)	71.4 (161)	59.6 (161)
Kimi-K3	87.6 (1064)	69.8 (1064)	75.0 (32)	73.3 (161)	57.8 (161)
GPT-5.6-sol	88.8 (1064)	71.9 (1064)	90.6 (32)	72.7 (161)	59.6 (161)
Gemini-3.5-flash	87.8 (1064)	72.1 (1064)	93.8 (32)	71.4 (161)	59.6 (161)
Claude-opus-4.8	87.7 (1064)	71.7 (1064)	78.1 (32)	72.0 (161)	60.9 (161)

Appendix

Table 17: Occupation prefix accuracy (percent), with eligible counts in parentheses. A hit requires the full prefix through that level.

System	Job $\rightarrow$ talent	Talent $\rightarrow$ job
Claude-opus-5	116/116	361/361
Kimi-K3	116/116	360/361
GPT-5.6-sol	116/116	361/361
Gemini-3.5-flash	116/116	361/361
Claude-opus-4.8	116/116	360/361
GLM-5.3	115/116	318/361

Appendix

Table 18: Groups satisfying the complete-permutation ranking protocol. Main nDCG@5 averages over all groups, with zero for groups failing the protocol.

	Reference-matching		Reference-nonmatching
System	$n$	Mean rubric	$n$	Mean rubric
Claude-opus-5	706	93.34	166	91.34
Kimi-K3	729	90.14	143	85.24
GPT-5.6-sol	698	93.18	174	91.93
Gemini-3.5-flash	658	90.32	214	86.70
Claude-opus-4.8	736	88.45	136	79.74

Appendix

Table 19: PJFA rubric conditional on parsed verdict agreement. Each system’s two subsets sum to 872 outputs; rubric scores are on a 0–100 scale.

System	Nonempty outputs	JSON parsed	Parse rate (%)	Verdict available (%)
Claude-opus-5	872	858	98.39	98.39
Kimi-K3	872	870	99.77	99.77
GPT-5.6-sol	872	872	100.00	100.00
Gemini-3.5-flash	872	856	98.17	98.17
Claude-opus-4.8	872	869	99.66	99.66
GLM-5.3	872	871	99.89	99.89

Appendix

Table 20: PJFA output parsing under the shared evaluation protocol. Verdict availability measures successful extraction through the benchmark JSON parser.

Comparator	Task	$n$	$C$	Difference	95% interval
GLM-5.3	JDG	941	941	+3.05	[+2.85, +3.25]
GLM-5.3	PJFA	872	872	+1.50	[+0.42, +2.59]
GLM-5.3	QP	714	714	-0.45	[-1.66, +0.75]
GLM-5.3	JC	1064	1064	+0.28	[-1.41, +2.02]
GLM-5.3	TC	161	161	-0.93	[-6.21, +4.04]
GLM-5.3	PJF	1386	874	+1.27	[-0.32, +2.85]

Appendix

Table 21: Paired differences against Claude-opus-5.

n

: common records/groups;

C

: observed-entity components. A dash denotes a single component.

Comparator	Task	$n$	$C$	Difference	95% interval
Kimi-K3	JDG	941	941	+1.19	[+0.98, +1.40]
Kimi-K3	PJFA	872	872	-1.43	[-2.41, -0.43]
Kimi-K3	QP	714	714	+0.36	[-0.66, +1.39]
Kimi-K3	JC	1064	1064	+0.88	[-0.69, +2.43]
Kimi-K3	TC	161	161	+0.00	[-3.42, +3.73]
Kimi-K3	PJF	1386	874	-0.97	[-2.69, +0.69]

Appendix

Table 22: Leading-system paired task differences.

n

: common records/groups;

C

: observed-entity components. A dash denotes one connected component.

Omitted family	Leading system	Five-family mean
Generation	Claude-opus-5	78.428
Extraction	Claude-opus-5	71.030
Classification	Claude-opus-5	75.194
Retrieval	Kimi-K3	76.770
STS	Claude-opus-5	73.414
Ranking	GLM-5.3	72.430

Appendix

Table 23: Leading five-family mean after omitting each capability family.

	JDG		PJFA		TC		TR
System	Public (500)	Public JD (441)	Derived (500)	Synthetic (372)	Public (91)	Synthetic (70)	Public (104)	Derived (80)
Claude-opus-5	43.46	48.48	57.41	73.44	40.66	97.86	45.55	86.53
Kimi-K3	42.65	46.87	58.95	74.72	41.21	97.14	42.86	86.53
GPT-5.6-sol	41.14	46.89	57.18	76.17	40.66	99.29	43.08	83.92
Gemini-3.5-flash	42.05	47.54	53.61	71.85	39.56	99.29	41.15	85.74
Claude-opus-4.8	41.89	46.45	58.93	75.09	42.86	97.14	40.09	86.91

Appendix

Table 24: Source scores for JDG, PJFA, TC, TR. Header counts give stratum sizes.

	SimT		SimJ		Dedup		J2T
System	Derived (500)	Synthetic (311)	Derived (540)	Synthetic (316)	Derived (616)	Synthetic (254)	Derived (110)	Synthetic (6)
Claude-opus-5	65.85	53.89	74.09	94.64	99.51	95.67	73.65	74.87
Kimi-K3	58.33	72.78	71.92	98.15	97.89	89.32	72.42	79.36
GPT-5.6-sol	64.14	56.95	72.77	99.07	99.03	93.69	73.05	81.55
Gemini-3.5-flash	67.90	72.81	74.95	91.97	97.08	94.88	74.16	76.03
Claude-opus-4.8	59.44	65.49	73.75	96.67	98.05	85.29	73.59	74.87

Appendix

Table 25: Source scores for SimT, SimJ, Dedup, J2T. Header counts give stratum sizes.

System	Behavior BA	Constructed BA	Behavior TPR	Behavior TNR
Claude-opus-5	52.69	92.59	72.0	33.4
Kimi-K3	54.20	94.44	74.3	34.1
GPT-5.6-sol	56.64	89.66	71.3	41.9
Gemini-3.5-flash	54.43	94.44	81.4	27.5
Claude-opus-4.8	52.01	93.52	75.1	28.9
GLM-5.3	54.75	92.75	74.9	34.6

Appendix

Table 26: PJF performance by construction stratum (percent). Behavior TPR/TNR use 478 positive/422 negative final labels.

Held-out entity	Text features	AUC	AUC SD	BA (%)
Resume	Resume + job	0.661	0.002	61.01
Resume	Resume only	0.546	0.005	54.26
Resume	Job only	0.671	0.006	63.28
Post	Resume + job	0.651	0.005	60.30
Post	Resume only	0.539	0.008	52.62
Post	Job only	0.666	0.005	62.72

Appendix

Table 27: Supervised PJF probe (

n=900

): AUC from held-out probabilities; BA at a 0.5 threshold.

Figure 8: Positive recall and negative rejection for (a) SimT (

n=811

) and (b) behavior-derived PJF (

n=900

). Dashed lines mark 50% balanced accuracy; numbers identify systems.

Task	Stratum and statistic	Total $n$	System range
SimT	Same-category negatives: TNR	167	58.08–98.20
SimT	Cross-category negatives: TNR	166	99.40–100.00
SimJ	Same-category final negatives: TNR	141	4.96–22.70
SimJ	Cross-category negatives: TNR	199	99.50–100.00
Dedup	Masked identity positives: TPR	250	93.60–100.00
Dedup	Identity snapshots: TPR	130	43.08–96.15

Appendix

Table 28: Score ranges across twelve systems by construction condition. Binary rows use final-label recall/rejection; QP uses span F1.

Task	Claude-opus-5	GPT-5.6-sol	Others’ mean	Others’ range
PJFA	73.44 (9)	76.17 (2)	71.82	47.49–77.06
TC	97.86 (4)	99.29 (1)	94.00	85.00–99.29
SimT	53.89 (12)	56.95 (10)	67.82	55.25–75.30
SimJ	94.64 (7)	99.07 (1)	94.52	89.71–98.15
Dedup	95.67 (1)	93.69 (3)	85.50	67.56–94.88

Appendix

Table 29: Controlled-synthetic scores of the two resume generators versus the other ten systems. Parentheses give ranks among all twelve systems, with ties.

Task	Source	$n$	Uniform	Input	TF-IDF	Model	$\Delta$
TR	Public	104	16.68	19.65	37.90	45.55	+7.66
TR	Derived	80	53.34	54.45	72.22	86.53	+14.31
J2T	Derived	110	43.10	39.92	56.97	74.16	+17.19
J2T	Synthetic	6	44.75	33.33	13.63	76.03	+62.40
T2J	Synthetic	361	36.86	39.80	89.05	97.19	+8.14

Appendix

Table 30: Baseline and model scores within each source. The model is the overall task leader: Claude-opus-5 for TR/T2J and Gemini-3.5-flash for J2T.

\Delta

: model minus TF-IDF.

Task	System	W/T/L	$\Delta$	95% interval
TR	Claude-opus-5	101/45/38	+10.55	[+7.71, +17.86]
J2T	Gemini-3.5-flash	63/39/14	+19.53	[+10.80, +28.60]
T2J	Claude-opus-5	75/276/10	+8.14	–

Appendix

Table 31: Paired gains over TF-IDF on all groups. W/T/L counts higher/equal/lower group scores at four decimals (0–1 scale). Intervals use observed-entity components; T2J has one component.

Property	External	Internal	Comparison
Job-classification records	1,064	10,599	18/18 classes; TV = 0.140
Talent-classification records	161	7,716	17/18 classes; TV = 0.140
Job-text mean / median chars	526 / 460	528 / 423	Mean difference 0.4%
PJF city coverage (%)	92.1	100	7.9 pp difference
PJF education coverage (%)	99.9	100	0.1 pp difference
PJF experience coverage (%)	97.9	100	2.1 pp difference

Appendix

Table 32: External and private-reference populations. TV measures occupation distribution distance; ranking pool means are J2T/T2J.

Figure 9: Occupation shares across eighteen categories. Categories below 3% in all four cohorts form Other. Tables 32 and 34 give cohort sizes and counts.

Statistic	Measurement	Sample count
Occupation proportions	First-level reference category, cate_labels[0] ; TV is half the summed absolute proportion differences.	Job: 1,064/10,599 Talent: 161/7,716
Job-text length	Character count of input.post_text in job classification.	1,064/10,599
PJF field coverage	Same regex/rule detector applied to resume input text (external: input.resume_info ) for city, education, and work-experience content.	1,386/880
Ranking pool size	Number of entries in input.candidates , summarized per group.	J2T: 116/123 T2J: 361/82

Appendix

Table 33: Measurement definitions for population comparisons. Sample counts are external/internal; ranking units are complete groups.

Occupation	Ext. $n$	Ext. (%)	Int. (%)	Ext. $n$	Ext. (%)	Int. (%)
	Job classification			Talent classification
Technology	133	12.50	26.50	24	14.91	28.82
Operations	241	22.65	18.87	31	19.25	15.98
General	183	17.20	14.33	28	17.39	14.67
Marketing	95	8.93	7.48	13	8.07	6.88
Product	74	6.95	6.36	13	8.07	6.74

Appendix

Table 34: Complete first-level occupation counts and shares. Counts are external records; percentages use each cohort’s denominator.

\dagger

marks the categories grouped as Other in Figure 9 : each is below 3% in all four cohorts.

System	Overall	Gen	Ext	Cls	Ret	STS	Rank
Kimi-K3	70.48	51.34	92.51	70.05	76.32	67.56	65.12
Qwen3.8-max	70.18	51.55	92.95	70.51	77.36	64.49	64.21
GLM-5.2	70.14	52.51	92.71	68.53	77.79	66.19	63.13
Kimi-2.6	69.68	51.00	90.54	69.83	75.77	65.97	64.99
DeepSeek v4-Pro	68.55	47.75	91.94	67.51	72.88	67.34	63.89
Internal solution	63.79	47.53	84.24	69.28	69.86	48.14	63.70

Appendix

Table 35: Private-reference capability scores; bold marks column maxima. The task-specific internal suite is excluded from shared-model correspondence.

	Gen		Ext	Cls			Ret	STS			Rank
System	JDG	PJFA	QP	JC	TC	PJF	TR	SimT	SimJ	Dedup	J2T	T2J
Kimi-K3	48.96	53.73	92.51	72.83	72.98	64.34	76.32	49.82	63.55	89.31	67.05	63.20
Qwen3.8-max	49.80	53.30	92.95	72.98	72.80	65.76	77.36	39.82	64.96	88.68	65.88	62.54
GLM-5.2	52.09	52.93	92.71	73.36	69.61	62.63	77.79	43.32	65.86	89.40	65.21	61.06
Kimi-2.6	51.47	50.52	90.54	74.27	70.36	64.87	75.77	59.06	57.63	81.23	67.53	62.45
DeepSeek v4-Pro	50.99	44.50	91.94	72.79	68.48	61.26	72.88	51.91	67.25	82.85	64.43	63.34

Appendix

Table 36: Task scores on the private operational reference.

View	Spearman $\rho$	Same / all	Opposite
Overall	1.00	10/10	0
Generation	0.60	7/10	3
Extraction	0.70	8/10	2
Classification	0.40	7/10	3
Retrieval	0.60	7/10	3
STS	0.20	6/10	4

Appendix

Table 37: Overall and category correspondence for the five shared systems.

Task	Spearman $\rho$	Same	Opposite	Tied
JDG	0.00	5	5	0
PJFA	1.00	10	0	0
QP	0.70	8	2	0
JC	0.10	5	5	0
TC	0.36	6	3	1
PJF	0.00	4	6	0

Appendix

Table 38: Task correspondence. Each task contributes ten comparisons.

Omitted capability	Spearman $\rho$
Generation	1.00
Extraction	0.90
Classification	0.90
Retrieval	1.00
STS	0.90
Ranking	1.00

Appendix

Table 39: Public–private ranking agreement among five shared models after omitting the same capability family from both collections. Each remaining family has equal weight.

		Public		Private
Omitted	Model	Mean	Rank	Mean	Rank
Generation	Kimi-K3	78.106	1	74.312	1
	Qwen3.8-max	76.632	2	73.904	2
	GLM-5.2	76.524	3	73.670	3
	Kimi-2.6	76.424	4	73.420	4
	DeepSeek v4-Pro	75.154	5	72.712	5

Appendix

Table 40: Five-capability means and model ranks underlying Table 39 . Means average two-decimal reported inputs and are displayed to three decimals to preserve differences introduced by averaging. Rank 1 is highest; equal sums receive average ranks.

Resource	Language / focus	Outputs / candidate conditions	Label evidence
ResumeBench ( Ling et al., 2025 )	Five languages, including Chinese	Resume PDF to schema-defined JSON	Curated templates; GPT-4o-generated JSON aligned with PDF layouts
JobResQA ( Carrino et al., 2026 )	Five languages, including Chinese	Short answers and explanations over synthetic resume–JD pairs	Linguist-authored QA; human-assisted translation
PJB ( Wang et al., 2026 )	Chinese retrieval	297 JD queries; fixed collection of 197,674 resumes	Doubao screening and Kimi relevance assessment; human checks on about 20% of queries
WorkRB ( De Lange et al., 2026 )	Multilingual work domain	Ranking and job-to-skill multilabel classification; ontology- and corpus-based candidate spaces	Task-specific datasets and occupational ontologies; extensible framework
EHR-Bench	Chinese recruitment	12 operations; generated text, typed spans, category paths, binary relations, retrieval, complete permutations	Source-aware label verification; assisted checks and expert judgments for constructed relations

Appendix

Table 41: Recruitment resources by language, outputs, candidate conditions, and label evidence. WorkRB coverage includes the accompanying toolkit ( TechWolf Research Team, 2026 ) .

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,401 papers

WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain
Mar 17, 2026Matthias De Lange, Warre Veys, Federico Retyk +16AI and Labor Recommender Systems
TechWolf Ghent, Belgium · Avature Barcelona, Spain · Malt Paris, France +8
LLMRec: Benchmarking Large Language Models on Recommendation Task
Aug 23, 2023Junling Liu, Chao Liu, Peilin Zhou +8Large Language Model-Based Recommendation Recommender Systems
Alibaba Group · Ant Group · The Hong Kong University of Science and Technology (Guangzhou) +5
Mira-Embeddings-V1: Domain-Adapted Semantic Reranking for Recruitment via LLM-Synthesized Data
Apr 20, 2026Zhaohua Liang, Zhilin Wang, Renjie Cao +1Learning to Rank Semantic Search
OpenJobs AI · San Francisico, CA, USA

Mar 17, 2026J/K move · Enter open · S save

WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain

Matthias De Lange, Warre Veys, Federico Retyk, Daniel Deniz+15

TechWolf Ghent, Belgium · Avature Barcelona, Spain · Malt Paris, France+8

AI and Labor Recommender Systems

Open paper AskPDF arXiv

Today's evolving labor markets rely increasingly on recommender systems for hiring, talent management, and workforce analytics, with natural language processing (NLP) capabilities at the core. Yet, research in this area remains highly fragmented. Studies employ divergent ontologies (ESCO, O*NET, national taxonomies), heterogeneous task formulations, and diverse model families, making cross-study comparison and reproducibility exceedingly difficult. General-purpose benchmarks lack coverage of work-specific tasks, and the inherent sensitivity of employment data further limits open evaluation. We present WorkRB (Work Research Benchmark), the first open-source, community-driven benchmark tailored to work-domain AI. WorkRB organizes 13 diverse tasks from 7 task groups as unified recommendation and NLP tasks, including job/skill recommendation, candidate recommendation, similar item recommendation, and skill extraction and normalization. WorkRB enables both monolingual and cross-lingual evaluation settings through dynamic loading of multilingual ontologies. Developed within a multi-stakeholder ecosystem of academia, industry, and public institutions, WorkRB has a modular design for seamless contributions and enables integration of proprietary tasks without disclosing sensitive data. WorkRB is available under the Apache 2.0 license at https://github.com/techwolf-ai/WorkRB.

Abstract

LLM Evaluation

Explore similar work

WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain

LLMRec: Benchmarking Large Language Models on Recommendation Task

Mira-Embeddings-V1: Domain-Adapted Semantic Reranking for Recruitment via LLM-Synthesized Data