cs.CYAug 19, 2026
SaveCentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks
Organizations: Data Innovation and AI Lab (DIAL) at the Haas School of Business, UC Berkeley · UC Berkeley · UC Berkeley and NBER
Abstract
Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.
Figures & tables
Figure 1: Methodology pipeline. The framework evaluates models in two usage modes: automation, where focal models directly solve tasks, and augmentation, where focal models provide an assistance text to a fixed worker model. Outputs are evaluated through rubric-guided pairwise comparisons and aggregated into model rankings by task and usage mode.
| Task | Source | Task type | Deliverable |
|---|---|---|---|
| Menu Planning | GDPval | Structured planning | Seven-day constrained meal plan |
| Travel Planning | GDPval | Structured planning | Tokyo itinerary under fixed budget |
| Market Trends Analysis | GDPval | Professional / analytical | Client brief on natural gas trends |
| Operations Research Analysis | Anthropic Economic Index | Professional / analytical | Executive logistics optimization memo |
| Tax Preparation | Designed task | Professional / analytical | Tax discrepancy review and correction guidance |
| Counseling | Anthropic Economic Index | Human-facing interactive | Supportive counseling-style response |
Figure 2: Tasks included in the benchmark. The figure reports each task’s source, broad task type, and required deliverable. Colors distinguish structured-planning tasks (yellow), professional or analytical tasks (green), and human-facing interactive tasks (purple). Tasks were drawn from GDPval and the Anthropic Economic Index, except for tax preparation, which was designed by the authors to provide a rule-based professional task with verifiable discrepancies. Full task prompts and evaluation rubrics are provided in Appendix B .
Figure 3: Prompt and rubric design. Each task prompt specifies observable deliverable requirements, which are mirrored by task-specific micro-rubrics. General rubric dimensions are held constant across tasks, while task-specific dimensions vary by domain. In augmentation mode, the assistance text creation prompt constrains assistant models to provide process guidance rather than the final deliverable.
Figure 4: Illustrative task prompt, evaluation rubric, and assistance-text instruction. The travel-planning example shows how observable task requirements (top) map onto task-specific rubric dimensions (middle). In augmentation mode, the assistant additionally receives the universal assistance text creation prompt shown above (bottom), which requires process-oriented guidance while prohibiting completion of the final deliverable. The complete prompts and rubrics are provided in Appendix B .
Figure 5: Rubric-guided pairwise evaluation pipeline. Candidate outputs are anonymized, randomly paired, and evaluated by LLM judges using task-specific and general rubric dimensions. Pairwise choices are converted into win rates, aggregated across eligible judges and repeated runs, and reported alongside rubric scores and rationales as an audit trail.
Figure 6: Rubric-guided scoring. Judge rationales explain the pairwise choice made. The models used in this example are GPT-4.1 and DeepSeek-V3.1 in augmentation mode for the counseling task. The corresponding outputs are in Appendix G .
Figure 7: Automation rankings by task, averaged across ten independent runs. For each task, we first average each model’s within-run rank across the ten runs and then rank these mean values to produce the displayed rank of ranks (1 = best). Rankings are computed using unrounded mean ranks. Exact ties receive the same rank, with the subsequent rank skipped (e.g., 1, 1, 3). Models are ordered from newest to oldest from left to right. Lower ranks indicate better performance. Mean ranks and standard errors are reported in Appendix C , Figure 10 .
Figure 8: Augmentation rankings by task, averaged across ten independent runs. For each task, we first average each model’s within-run rank across the ten runs and then rank these mean values to produce the displayed rank of ranks (1 = best). Rankings are computed using unrounded mean ranks. Exact ties receive the same rank, with the subsequent rank skipped (e.g., 1, 1, 3). Models are ordered from newest to oldest from left to right, with GPT-3.5-Turbo (plain), the unaided worker baseline, shown at far right. Lower ranks indicate better performance. Mean ranks and standard errors are reported in Appendix C , Figure 11 .
Figure 9: Model-level comparison of average automation rank and average augmentation rank (assistant models only) . Each point represents one assistant model’s average rank across seven tasks and ten independent runs. Ranks are inverted so that higher values indicate better performance. With nine assistant models, reverse rank is defined as , where is the model’s mean rank in the corresponding usage mode. The horizontal axis reports automation performance and the vertical axis reports augmentation performance. The unaided worker baseline is excluded.
| Task | Spearman | -value |
|---|---|---|
| Tax Preparation | 0.85 | 0.004 |
| Menu Planning | 0.64 | 0.066 |
| Tutoring Session Planning | 0.50 | 0.166 |
| Counseling | 0.25 | 0.516 |
| Operations Research Analysis | 0.17 | 0.668 |
| Market Trend Analysis | 0.10 | 0.798 |
Table 1: Task-level correlation between automation and augmentation rankings. Spearman correlations are computed across the nine assistant models using each model’s mean rank over ten runs. Positive values indicate similar cross-mode ordering; values near zero indicate little correspondence. -values are descriptive because each task contains only nine models.
| Task, assistant (rank) | Assistance-text excerpt | |
|---|---|---|
| Trait 1: Specificity beyond the prompt (contingency) | ||
| Good | Market trends, GPT-5-Mini (1) | “For each trend, plan 2 to 3 succinct bullets: core observation, primary driver(s), and potential market implication…a short preface about methodology and data vintage.” |
| Bad | Market trends, GPT-4.1 (9) | “Ensure you understand the expectation to discuss how various factors (supply, demand, weather, storage, LNG exports, infrastructure, policy) interact…” (restates the prompt’s factor list) |
| Trait 2: Explicit structure and ordering | ||
| Good | Travel planning, Gemini-3.1-Pro (3) | Enforces assumptions and clarifying questions “ before the provisional itinerary,” with named sections in explicit order: Introduction Budget Breakdown Daily Itinerary Conclusion. |
| Bad | Travel planning, GPT-O4-Mini (9) | “Determine sequence and hierarchy of information for clarity and flow”; instructs worker to “populate each section with neutral placeholders… [Day 1: transportation placeholder] ”; no ordering is prescribed. |
Table 2: Illustrative higher- vs. lower-ranked assistance texts for each scaffold-quality characteristic. Rows are color-coded by trait; ranks are the aggregate augmentation rank within the task run. Excerpts are drawn from the assistance texts and lightly trimmed.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Provider | Automation | Augmentation | Judge |
| GPT-3.5-Turbo | OpenAI | ✓ | Worker | |
| Claude-Opus-4.8 | Anthropic | ✓ | ✓ | ✓ |
| Claude-Sonnet-4.6 | Anthropic | ✓ | ✓ | |
| DeepSeek-V3.1 | DeepSeek | ✓ | ✓ | ✓ |
| Gemini-3.1-Pro | ✓ | ✓ | ✓ | |
| GPT-4.1 | OpenAI | ✓ | ✓ | ✓ |
Table 3: Models used in the evaluation and their assigned roles. All focal models serve as direct solvers in the automation regime and as planning assistants in the augmentation regime. GPT-3.5-Turbo serves as the fixed worker in augmentation mode rather than as a focal model. Judges follow a leave-family-out constraint: each judge is excluded from evaluating outputs of models from its own provider family.
Figure 10: Automation mean ranks by task with standard error across ten independent runs.
Figure 11: Augmentation mean ranks by task with standard error across ten independent runs.
| Task | Automation | Augmentation |
|---|---|---|
| Counseling | GPT-OSS-120B | GPT-4.1 |
| Market Trends | Claude-Opus-4.8 | GPT-O4-Mini |
| Menu Planning | GPT-5-Mini | GPT-5-Mini |
| Operations Research | Gemini-3.1-Pro / GPT-OSS-120B | GPT-3.5-Turbo (plain) |
| Tax Prep | GPT-5-Mini | GPT-3.5-Turbo (plain) |
| Travel Agent | GPT-5-Mini | GPT-3.5-Turbo (plain) |
Table 4: Best-performing model by task and regime , based on lowest mean rank across ten independent runs.
Figure 12: Average general-rubric scores in the augmentation mode, aggregated across tasks and judges. Scores are shown for instruction following, accuracy and specificity, practical usefulness, organization and readability, and tone/audience fit. Profiles are relatively flat across dimensions for most models, indicating that ranking differences are not driven by specialization on a single general rubric axis.
Figure 13: Validation of the rubric-guided pairwise judging procedure across ten independent runs. Panel A compares each model–task–mode–judge cell’s mean rubric score with its pairwise win rate ( ; cells). Panel B reports, for each judge, the percentage of non-tied comparisons in which the final pairwise choice agrees with the option receiving the higher average rubric score. Agreement ranges from 98.9% to 100.0% across judges.
Figure 14: Inter-judge reliability across ten independent runs. Panel A reports raw agreement and Krippendorff’s nominal alpha overall and separately for augmentation and automation. Raw agreement is the share of eligible judge-pair decisions selecting the same winner; alpha adjusts for chance agreement and accommodates missing ratings induced by leave-family-out evaluation. Panel B reports alpha by task and usage mode. Only comparisons evaluated by at least two eligible judges are included.
Figure 15: Criterion-validity checks using automation outputs across ten independent runs. Panel A reports mean within-family rank for GPT-family models judged by Claude-Opus-4.8; the judge recovers the prespecified release-order gradient in 89% of ordered comparisons. Panel B reports mean within-family rank for Claude-family models judged by GPT-4.1; Claude-Opus-4.8 is preferred in 62.9% of task–run comparisons. Error bars show one standard deviation across task–run cells. Lower ranks indicate better performance.
More in this topic
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,132 papers