PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?

Mohammad Reza Rezaei; Dingxian Wang; Brett Levert; Celeste Fejzo; Andrew Rabinovich; Zhao Chen

cs.AINov 3, 2026

PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?

Authors: Mohammad Reza Rezaei, Dingxian Wang, Brett Levert, Celeste Fejzo, Andrew Rabinovich, Zhao Chen

Organizations: Upwork Inc.

Abstract

Before an AI agent takes on a paid job, someone has to judge whether it is likely to succeed. We test whether frontier language models can make this call before any attempt. We propose PAYWORTHY which contains 1{,}494 real freelance job postings, sampled from 18.8M postings made between 2020 and 2026 on a freelancing platform. An agent attempts each job from the posting alone in a single run, and an LLM judge acting as the client decides whether to accept the work. We ask 18 models to forecast how likely the agent is to succeed and measure how well each model's forecasts rank the accepted jobs above the rejected ones (AUC, where 0.5 is a random ranking). The answer depends on how the agent is described. Told what the agent cannot do, every model ranks the jobs well (AUC 0.66--0.86). Told instead that it is a generic agent with a browser and code execution, the same models rank the same jobs about as well as a random ordering (0.45--0.56). These results hold under judges from four other model families (Claude, GLM, Kimi and Qwen), and the informed forecasts also rank the verdicts of two human raters. Code is available at https://github.com/upwork/Payworthy

Figures & tables

Figure 1: PayWorthy in brief. (1) A real job posting. (2) Before any attempt, a model forecasts the probability

\hat{p}

that the agent will succeed. (3) The agent attempts the job from the posting alone, and an LLM judge acting as the client decides whether to accept the work. (4) We measure how well

\hat{p}

ranks accepted jobs above rejected ones (AUC; 0.5 is a random ranking). When told the agent’s limits, the 18 forecasters reach AUC 0.66–0.86; when asked about a generic tool-using agent, they stay near 0.5.

Figure 2: Choosing jobs by forecast: the acceptance rate among the top

k\%

of jobs, ranked by each of the 18 forecasters. The band covers 95% of 2,000 random selections of the same size. At

k=20\%

, limits forecasts reach 65–92%, generic forecasts 41–54% (below the band for 15 of 18), and a cross-fitted classifier on posting metadata 77% (§ 5.2 ).

Benchmark	Tasks	Domain	Grader	Prices	Sampled	Forecast
SWE-bench ( Jimenez et al., 2023 )	2,294	Python repos	unit tests	no	no	no
$\tau$ -bench ( Yao et al., 2024 )	165	retail, airline	DB state + rules	no	no	no
SWE-Lancer ( Miserendino et al., 2025 )	1,488	software	E2E tests	yes	no	no
GDPval ( Patwardhan et al., 2025 )	1,320	44 occupations	expert comparison	no	no	no
RLI ( Mazeika et al., 2025 )	240	remote freelance	expert review	yes	no	no
UpBench ( Yi et al., 2025 )	322	freelance jobs	expert rubrics	yes	no	no

Table 1: PayWorthy among benchmarks of agentic and economically valuable work. “Grader”: what decides success. “Prices”: tasks carry real posted prices. “Sampled”: items are drawn at random or by stratified sampling from posted work, with no person choosing or screening them ( PayWorthy applies one fixed automatic filter; § 3.1 ). “Forecast”: models predict success before the attempt.

Figure 3: Corpus and priced demand. (a) Posts by era (18.8M in total). (b) Priced demand in billions of dollars (

69.1B posted,

20.4B realized in total). Realized demand counts only postings with at least one awarded contract.

Figure 4: Agent outcomes on PayWorthy (solver claude-opus-5; judge gpt-5 reading in full, three votes per job). (a) Share of jobs meeting each of the judge’s criteria (dark) and the same share weighted by contract value over priced jobs (light; mostly reflects pay type, Appendix B ). Each criterion is a separate majority vote, so the shares need not fall from one criterion to the next. (b) The judge’s failure reasons for the 682 rejected jobs (counts in parentheses).

Figure 5: Can models tell in advance which jobs the judge will accept? Each forecaster rates every posting with the generic prompt (open circles) and the limits prompt (filled circles), compared with the judge’s verdict on the attempt (headline label); forecasters are sorted by limits AUC within each group. (a) AUC with 95% bootstrap CI; 0.5 is a random ranking. Every forecaster’s gain from the limits prompt excludes 0 (paired bootstrap). (b) Mean forecast against the acceptance rate (dashed line, 0.545 on these postings); the multiplier is the mean generic forecast divided by that rate, so values above 1 are overclaims. (c) 10-bin expected calibration error. Averaging rows average the generic forecasts of the seven base forecasters or of the fifteen OpenAI and Anthropic forecasters. Baselines use no forecast and are 5-fold cross-validated, except log posted budget, which ranks the 350 hourly jobs without a stated rate as the cheapest (among priced jobs it reaches 0.58); description embedding is a logistic regression on text embeddings of the posting.

Category	Jobs	Share	Accepted
Design & Creative	385	25.8	42.9
Web, Mobile & Software Dev	248	16.6	66.9
Engineering & Architecture	204	13.7	55.9
Writing	136	9.1	55.9
Data Science & Analytics	109	7.3	67.9
Accounting & Consulting	84	5.6	65.5

Table 2: Jobs per top-level category in PayWorthy , their share of the 1,494 jobs and the share the judge accepts (headline label), both in percent, with the same split by pay type.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Step	Left
Corpus, 2020 to mid-2026 (2,389 daily files)	18.77M
1st and 15th of each month (158 days, 79 months); up to 15 per L3 category (205) per day, in file order; descriptions stored to 2,000 characters	286,777
Stratified draw: L1 $\times$ month (948 cells), proportional, $\geq$ 1 per cell, de-duplicated on title and description	4,000
Description over 150 characters; shuffled once (seed 17)	3,408
Walked in order until 1,500 kept	2,614
excluded: fewer than four votes for a defined deliverable	$-$ 1,114

Appendix

Table 3: From the corpus to PayWorthy (§ 3.1 ). Each row is the count left after that step.

Figure 6: Label noise on 400 items (headline judge). (a) Share pay-worthy with the brief fixed and the work varied; in parentheses, the share among items whose first attempt passes / fails. (b) AUC against one attempt; the oracle is a beta-binomial predictor knowing each item’s true pass rate.

Check	$n$	Result
The raters on the judge’s accepts
Produced / on-spec: raters (judge)	149	99.3 / 82.6% (100 / 94.0%)
Pay-worthy, raters	149	34.9% [28.2, 42.3]
Raters’ rejections labelled missing information	97	75.3%
What the job needs
Raters’ pay-worthy by the needs label: text only / client files	60 / 53	53.3 / 17.0%

Appendix

Table 4: Human raters on 149 items the headline judge accepted (Appendix E ). Shares are of the rated items unless noted; brackets are 95% bootstrap CIs (resampling within category for the pay-worthy rate). AUCs are against the raters’ pay-worthy call, and ranges are across the eighteen forecasters;

\Delta

is a paired gain of the limits over the generic prompt.

Identifier	Role	Calls
gpt-5	F, J	09-20–10-11
gpt-5-mini	F, C	09-20–10-11
gpt-5-nano	F	09-21–10-11
gpt-4.1	F, A	09-21–10-11
gpt-4o	F	09-21–10-11
o3	F	09-21–10-11

Appendix

Table 5: Models by the identifier sent to the provider, their roles (F forecaster, S solver, J judge, C the tool agent’s simulated client, A annotation of surrogates or reasons, E embeddings) and the first and last day we called them (2026). The text writes version separators as dots and drops snapshot dates: glm-5p3 is glm-5.3, qwen3p8-max qwen3.8-max, claude-opus-5-5 claude-opus-5.5, and claude-sonnet-4-5-20250929 and claude-haiku-4-5-20251001 are claude-sonnet-4.5 and claude-haiku-4.5.

Axis	Question put to the model
Context dependence	Needs the client’s private assets, brand voice, prior work, tacit taste
Verifiability	Can success be judged from the posting alone
Autonomy horizon	Steps before a human must intervene
Tool/credential surface	Needs accounts, payments, physical access
Deliverable modality	Emittable artifact vs. ongoing service

Appendix

Table 6: Annotation axes elicited alongside each forecast (Task B).

Predictor	$\hat{\beta}$	$p$
Log posted budget	$-0.25$	0.003
Hourly pay	$+0.60$	$<$ 0.001
Hourly, no stated rate	$-0.14$	0.088
Log description length	$+0.11$	0.078
Skill count	$+0.03$	0.570

Appendix

Table 7: Posting features in a standardized logistic regression of the headline label on the 1,474 PayWorthy jobs that gpt-5-mini annotated. The fit also includes gpt-5-mini’s five annotation axes (Table 6 ) as controls. Log budget is taken over priced jobs; the hourly jobs without a stated rate get the priced mean and their own indicator.

Figure 7: The claude-opus-5 deliverables graded by judges from five frontier model families (three votes each, same prompt; claude-sonnet-5 with thinking disabled), their majority, and the items on which all five agree. (a) Share accepted and Cohen

\kappa

against the headline label. (b) Range of forecaster AUC against that label over the eighteen forecasters, generic (light) and limits prompt (dark); dashed: random ranking (0.5).

Figure 8: Generic and limits forecasts against the outcomes of two agents on the same 500 jobs: the PayWorthy agent, and the tool-using agent with a simulated client that the generic prompt describes. Dots: one forecaster each (18 per prompt); diamonds: mean AUC. The two agents’ verdicts agree at Cohen

\kappa=0.29

. The interaction is the generic prompt’s advantage on the tool agent minus the limits prompt’s, with a 95% bootstrap CI over jobs.

Figure 9: Ablating the limits prompt (four forecasters, headline label, 95% paired-bootstrap CIs). (a) AUC change when one limit is dropped from the full prompt. (b) AUC gain over the generic prompt when only one limit is stated; dashed lines mark each forecaster’s full-prompt gain.

Check	$n$	Result
Anchor: real submissions to 322 jobs
Paid work, opening / full	256 / 255	19.9% [15.2, 24.6] 16.5% [12.2, 20.8]
claude-opus-5, opening / full	321 / 320	19.0% [14.6, 23.4] 27.8% [22.8, 32.8]
AUC, paid vs. the 10 rejected submissions, opening / full	256 / 255	0.63 [0.53, 0.69] 0.60 [0.47, 0.70]
Reading length: PayWorthy
Pay-worthy rate, full (headline)	1,494	54.4% [51.9, 57.0]

Appendix

Table 8: External anchor and reading length: the PayWorthy judge (gpt-5, three votes) on real submissions that a client paid for and on claude-opus-5’s attempts at the same jobs, reading only the opening or the whole deliverable. Share judged pay-worthy unless noted; brackets are 95% bootstrap CIs.

Label	Rate	$\kappa$	Generic	Limits	Gain	Text
gpt-5 (headline)	54.4	–	.45–.56	.66–.86	.20–.40	.82
gpt-5, opening	25.9	.42	.44–.59	.64–.81	.18–.33	.73
glm-5.3	53.5	.75	.44–.57	.69–.87	.25–.42	.81
claude-sonnet-5	32.6	.52	.43–.60	.70–.86	.22–.40	.80
qwen3.8-max	47.1	.72	.43–.58	.70–.88	.25–.42	.84
kimi-k3	57.6	.76	.46–.58	.69–.87	.23–.41	.81

Appendix

Table 9: Forecasts against eight labels of the same deliverables: gpt-5 reading the whole deliverable (headline) or the opening only, four judges from other model families reading in full, the majority of the five full-reading judges, and the 978 items on which all five agree. Generic and limits: AUC range over the eighteen forecasters; gain: range of paired gains of the limits over the generic prompt (every 95% CI excludes 0; lowest lower bound

+0.147

). Text: 5-fold cross-validated logistic regression on text-embedding-3-small embeddings of the title, description and skills.

\kappa

is against the headline label (on the unanimous items it is 1 by construction).

Check	$n$	Result
Selection and posting features
Own annotation left out of the PayWorthy selection (base seven)	1,384–1,425	AUC 0.456–0.557 (full rule 0.456–0.558)
Reweighted to corpus shares of postings / dollars: by category; by cluster	1,494	54.4% / 57.6%; 51.0% / 56.3% (as sampled 54.4%)
Spearman of contract value with outcome, priced jobs: fixed-price / hourly	754 / 390	$-0.05$ / $+0.20$
Spearman with outcome: log description length / skill count	1,494 / 1,429	$-0.01$ / $+0.08$
Annotations refit on outcomes (cross-fitted)

Appendix

Table 10: Checks on selection, posting features, annotation refits and surrogates, on the 1,494 PayWorthy items (headline label). AUC ranges are across forecasters.

Check	$n$	Result
Generic forecasts with missing-information rejections set aside	1,223	AUC 0.521–0.630; mean 0.584 [0.547, 0.618]
Missing-information rejection rate, short / long third of briefs	1,489	20.9% / 15.1%
The needs label, annotated by claude-sonnet from the posting: text only / client files / client-system access / a medium the agent cannot produce	1,494	share 25 / 43 / 18 / 14%; pay-worthy 82.3 / 41.8 / 63.7 / 32.1%

Appendix

Table 11: What the posting needs and missing information, on the 1,494 PayWorthy items (headline label). The AUC range is across forecasters; “mean” is the mean generic forecast of the fifteen OpenAI and Anthropic forecasters; brackets are 95% bootstrap CIs.

Check	$n$	Result
Separating self-contained jobs from jobs needing the client’s files: generic / limits forecasts (outcome 0.70)	999–1,016	AUC 0.41–0.57, 7 of 18 CIs below 0.5 / 0.69–0.87, all CIs above 0.5
Top fifth by each generic forecast (ties broken at random, mean of 20 draws): share needing the client’s files / a medium the agent lacks (all jobs 43 / 14%); acceptance of its client-file picks (all client-file jobs 42%); share of the top fifth that the same forecaster’s limits forecast puts in its top fifth / bottom half	1,472–1,494	51–62% (median 58) / 0–5% (median 3); 26–40% (median 31); 23–53% (median 38) / 21–54% (median 37)
Forecast (logit) explained by posting size (budget, description length, skills, pay type, tier) / needs label / own five annotations, cross-validated $R^{2}$ , medians	1,472–1,494	generic $\leq$ 0.03 (max) / 0.21 / 0.45; limits $\leq$ 0.01 (max) / 0.34 / 0.25
One-line reasons (gpt-5, gpt-5-mini, claude-sonnet-4.5, claude-opus-5; 300 items of the 400; cues tagged by gpt-4.1): share naming missing client material as a risk; mean forecast when they do / do not	1,199 per prompt	generic 37%: 0.50 / 0.49; limits 61%: 0.15 / 0.20
The same four with the reason field: AUC (without the field)	299–300	generic 0.44–0.59 (0.44–0.54); limits 0.79–0.88 (0.80–0.88)

Appendix

Table 12: What the forecasts track, on the PayWorthy items (headline label; eighteen forecasters unless noted). AUC ranges are across forecasters.

Check	$n$	Result
The needs label alone, cross-fitted	1,494	AUC 0.69; below 17 of 18 limits forecasts
Hand-written rule on the needs label, no labeled jobs: low if the job needs the client’s files / low unless self-contained	1,494	AUC 0.61 [0.58, 0.63] / 0.64 [0.62, 0.66]
Forecast minus the rule (low unless self-contained), paired: generic / limits	1,472–1,494	$-0.19$ to $-0.08$ , all CIs below 0 / $+0.02$ to $+0.22$ , 17 of 18 CIs above 0 (client-files rule: $+0.05$ to $+0.25$ , all CIs above 0)
AUC within jobs of the same needs label: generic / limits	1,472–1,494	0.39–0.49, 10 of 18 CIs below 0.5 / 0.60–0.83, all CIs above 0.5
$\Delta$ within jobs needing the client’s files / system access	633 / 256	$+0.21$ to $+0.46$ / $+0.22$ to $+0.36$ ; all CIs exclude 0
$\Delta$ within self-contained jobs / jobs needing a medium the agent lacks	366 / 215	$+0.10$ to $+0.36$ , all CIs exclude 0 / $-0.00$ to $+0.23$ , 12 of 18 CIs exclude 0

Appendix

Table 13: Checks on the limits prompt, on the 1,494 PayWorthy items (headline label; eighteen forecasters unless noted). AUC ranges are across forecasters; brackets are 95% bootstrap CIs;

\Delta

is a paired gain of the limits over the generic prompt.

Check	$n$	Result
Only deliverables written in a single call	1,158	generic AUC 0.47–0.56, limits 0.66–0.87; $\Delta$ $+0.18$ to $+0.40$ , all CIs exclude 0
Haiku 4.5 passes, within the jobs the benchmark agent passes (400-item subset): limits / generic	223–226	AUC 0.63–0.74, all CIs above 0.5 / 0.47–0.63, 7 of 18
claude-opus-5 on its own attempts vs. Fable’s and Haiku’s, double-centred AUC residual: limits / generic	395–398	$-0.011$ [ $-0.022$ , 0.001] / $-0.020$ [ $-0.034$ , $-0.006$ ]
Mean of two attempts predicting the third: observed / implied by the beta-binomial fit; its oracle AUC (item bootstrap)	399	0.944 / 0.945; 0.99 [0.98, 0.99]
Mean over all eighteen, items resampled jointly (items every forecast covers)	1,449	generic AUC 0.51, limits 0.80; $\Delta$ $+0.30$ [0.27, 0.32]; mean $\Delta$ by model family $+0.26$ to $+0.31$
Limits forecast recalibrated (Platt) on 100 random labeled jobs, scored on the rest (20 splits)	100 / rest	ECE 0.05–0.11 (uncalibrated 0.22–0.47); accuracy at 0.5: 0.63–0.78 (always accept 0.54)

Appendix

Table 14: Checks on the protocol and on dependence, on the PayWorthy items (headline label; eighteen forecasters). AUC ranges are across forecasters; brackets are 95% bootstrap CIs;

\Delta

is a paired gain of the limits over the generic prompt.

Setting	Agent	Forecasts	Label	Items	Main result
Benchmark (§ 5 )	claude-opus-5; posting only, one response, named files	generic, limits (18 forecasters)	gpt-5, full reading, 3 votes (headline)	1,494	generic 0.45–0.56; limits 0.66–0.86
Ablation (§ 5.1 )	same	limits with one limit dropped or alone; placebo (4 forecasters)	headline	1,494	L6 and L1 carry most of the gain
Tool agent (App. H )	claude-opus-5 with tools and a simulated client	generic, limits	gpt-5, also sees the client’s facts and exchange	500	generic near random; limits 0.56–0.69
Label noise (§ 5.3 )	benchmark agent, two repeat attempts; a weaker solver; altered work	best limits	headline judge, fresh votes	400	attempts $\kappa$ 0.81, votes 0.87; ceiling 0.94 (model-based 0.99) vs. 0.86
Other labels (§ 5.4 )	benchmark agent	generic, limits	gpt-5 opening only; four other judges; their majority; unanimous items	1,494 (978)	generic 0.43–0.61; limits 0.64–0.95
Other solvers (§ 5.4 )	claude-fable-5-1; claude-haiku-4.5; same harness	generic, limits	headline judge	400	generic 0.42–0.61; limits 0.68–0.87

Appendix

Table 15: Settings at a glance: which agent, forecast prompt and label each result uses. “Limits” and “generic” are the two Task B prompts (§ 5 , Appendix F ); AUC ranges are across forecasters.

Language Model Performance Prediction

See all 41 papers

AI evaluators favor job applications written by certain AI models, independent of candidate quality
Oct 30, 2026Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu +1Algorithmic Fairness Language Model Bias Evaluation
Universität Zürich, Zurich, Switzerland · ETH Zürich, Zurich, Switzerland
Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement
Oct 1, 2026Harry Lyu, Neil ThompsonLLM-as-a-Judge LLM Evaluation
Massachusetts Institute of Technology
JobBench: Aligning Agent Work With Human Will
May 25, 2026Yuetai Li, Yichen Feng, Zhangchen Xu +21Computer-Use Agent Benchmarks LLM Agent Evaluation
University of Washington · 10Bake AI · University of California, Santa Barbara +10

Oct 30, 2026J/K move · Enter open · S save

AI evaluators favor job applications written by certain AI models, independent of candidate quality

Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu, Anikó Hannák

Universität Zürich, Zurich, Switzerland · ETH Zürich, Zurich, Switzerland

Algorithmic Fairness Language Model Bias Evaluation Bias in LLM-as-a-Judge Language Model Auditing

Open paper AskPDF arXiv

Job applicants increasingly use large language models (LLMs) to write cover letters, while employers deploy LLMs to screen them. In this all-AI setting, we ask whether the choice of model affects hiring outcomes independently of candidate qualifications. We introduce a multi-model auditing framework pairing eleven LLM writer agents with nine API-hosted evaluator agents across ten job descriptions and fifty candidates per job. Four findings emerge. First, we identify a generative preference hierarchy: a consistent global ranking of writer models by evaluator acceptance, in which locally hosted models are systematically disadvantaged regardless of their license, compounded by self-preference bias where evaluators favour their own outputs. Second, evaluator models differ substantially in how much bias they introduce based on applicants' tool choice, offering actionable guidance for fairer screening design. Third, this hierarchy has measurable competitive consequences: in the most adverse pairing, approximately two (1.8) out of twenty-five qualified candidates are filtered out purely due to model mismatch. Fourth, a common mechanism explains the patterns above: writer preference and self-preference trace to a combination of how much each writer leans on specific stylistic cues and how strongly evaluators, often unanimously, reward or penalise those same cues. A further check finds little evidence that an unmeasured "writer fingerprint" remains once these same features are accounted for. We discuss implications for algorithmic fairness in employment, regulation of AI-assisted screening, and the design of auditing frameworks that can be reapplied as the model landscape evolves.

Abstract

Language Model Performance Prediction

Explore similar work

AI evaluators favor job applications written by certain AI models, independent of candidate quality

Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

JobBench: Aligning Agent Work With Human Will