Human-AI Agreement

The field compares AI outputs with human judgments in settings such as peer review, survey coding, safety assessment, clinical evaluation, and visual rating. Approaches include inter-rater reliability and rank correlations, LLM-as-judge prompts, expert-authored rubrics, confidence calibration, and interactive studies of delegation and correction. Agreement varies by task, model, and prompt, and rank-order similarity or matching broad decisions may not mean AI reproduces human scores or priorities. A central challenge is building evaluations and collaboration methods that account for human disagreement and support calibrated reliance on AI.