Counterfactual Pairwise Augmentation Improves Small Pointwise LLM Judges
Organizations: Huawei Technologies Ltd.
Abstract
Agentic systems increasingly gate individual steps with a pointwise judge that scores a response against a criterion. Small language models are attractive for this role because they can be served and specialized efficiently. Such a judge must locate the criterion's decision boundary, but pointwise labels under-determine that boundary in fine-tuning: labeled responses to different queries differ along many dimensions at once, so they do not reveal which features decide the label. We introduce CPA (Counterfactual Pairwise Augmentation), which makes the deciding difference explicit in training. For each pointwise item, CPA synthesizes and audits a counterfactual response to the same query intended to carry the opposite label. It frames the accepted pair as a pairwise task asking which of the two responses satisfies the criterion. The judge is fine-tuned jointly on both pointwise and pairwise datasets and deployed in pointwise mode only. A second-order analysis shows when this should outperform the prevailing strategy of merging separately trained pointwise and pairwise experts: merging moves every parameter direction toward the pairwise expert by the same fraction, whereas joint training can hold the pointwise optimum wherever the pairwise loss is flat and absorb its curvature where it is sharp. This is the regime that CPA's same-source, counterfactually minimal pairwise view is designed to produce. On separate Qwen3-8B LoRA adapters for six binary criteria, CPA improves criterion-averaged pointwise macro-F1 by 2.5 points over pointwise-only training and outperforms model merging on all six criteria at every tested merge weight. It also surpasses the strongest zero-shot frontier judge we evaluate, Gemini 3.6 Flash, by 11.8 points.
Figures & tables
| Criterion | Qwen3-8B baseline | Pointwise-only SFT | Pairwise-only SFT | Merged SFT ( Kim et al., 2024b ) | Joint SFT ( CPA ) |
| C1 AI-ness | 64.0 1.0 | 71.0 1.4 | 64.7 1.6 | 73.5 0.9 | 74.2 1.3 |
| C2 criminality | 67.5 0.8 | 89.4 0.6 | 68.3 1.0 | 89.8 0.6 | 91.3 0.6 |
| C3 safety leakage | 76.4 1.3 | 88.6 0.9 | 79.7 1.3 | 87.1 0.8 | 89.6 1.0 |
| C4 hallucination | 88.7 1.7 | 94.9 0.3 | 90.8 1.3 | 90.9 1.1 | 99.6 0.5 |
| C5 tool-param. | 71.6 1.5 | 88.8 0.4 | 70.4 1.6 | 86.6 1.0 | 89.9 0.7 |
| C6 correctness | 78.5 1.2 | 94.5 0.6 | 81.6 1.7 | 91.9 0.5 | 97.7 0.6 |
| Criterion | GPT 5.6 | Gemini 3.6 Flash | DeepSeek V3.2 | Qwen3-8B | Qwen3-8B CPA |
| C1 AI-ness | 35.9 | 53.9 | 30.1 | 64.0 | 74.2 |
| C2 criminality | 72.5 | 68.7 | 78.9 | 67.5 | 91.3 |
| C3 safety leakage | 83.7 | 75.4 | 84.0 | 76.4 | 89.6 |
| C4 hallucination | 87.5 | 91.0 | 80.9 | 88.7 | 99.6 |
| C5 tool-param. | 91.2 | 92.0 | 83.3 | 71.6 | 89.9 |
| C6 correctness | 85.7 | 90.6 | 73.7 | 78.5 | 97.7 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Criterion | |||
| C1 AI-ness | 69.7 | 71.4 | 73.4 |
| C2 criminality | 78.3 | 82.8 | 90.0 |
| C3 safety leakage | 82.7 | 83.0 | 87.3 |
| C4 hallucination | 87.7 | 88.3 | 89.9 |
| C5 tool-param. | 84.8 | 86.1 | 86.9 |
| C6 correctness | 84.8 | 88.6 | 92.3 |
| Setting | Value |
| Adapter (LoRA) | |
| Rank / scale | 32 / 64 (so ) |
| Dropout | 0.05 |
| Target modules | q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , down_proj |
| Trained biases | none |
| Task type | causal language modeling |
| Criterion | Pointwise | Pairwise | CPA |
| C1 AI-ness | 900 | 890 | 1,790 |
| C2 criminality | 1,232 | 1,166 | 2,398 |
| C3 safety leakage | 1,400 | 1,284 | 2,684 |
| C4 hallucination | 792 | 746 | 1,538 |
| C5 tool-param. | 1,392 | 1,220 | 2,612 |
| C6 correctness | 1,156 | 978 | 2,134 |
| Judge | Base model | Supervision signal | Scale | Objective |
| PandaLM [ Wang et al., 2024b ] | LLaMA | GPT-3.5 evaluations, reasons, and reference responses | 300K | SFT |
| Vicuna judge [ Zheng et al., 2023 ] | Vicuna-13B | Chatbot Arena single-turn human votes | 20K | SFT |
| Auto-J [ Li et al., 2024 ] | LLaMA-2-13B-chat | GPT-4 ratings and critiques over 58 scenarios | – | SFT |
| Prometheus [ Kim et al., 2024a ] | Llama-2-Chat-13B | Feedback Collection (GPT-4 rubrics, references, feedback) | 100K | SFT |
| JudgeLM [ Zhu et al., 2025 ] | 7B–33B | GPT-4 judgments and reasons on 100K seed tasks; swap augmentation and reference support | 100K | SFT |
| Self-Taught Evaluator [ Wang et al., 2024a ] | Llama3-70B-Instruct | Own judgments on synthetic contrasting pairs, kept where they match the intended label | – | iterative SFT |
| Criterion | Source | Ground-truth label |
| AI-ness | HC3 [ Guo et al., 2023 ] https://huggingface.co/datasets/Hello-SimpleAI/HC3 | Majority vote of five frontier LLMs on whether the response reads as AI-generated. |
| Criminality | Aegis 2.0 [ Ghosh et al., 2025 ] https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0 | Dataset is_safe annotation: safe , unsafe . |
| Safety leakage | Nemotron content safety [ NVIDIA, 2026 ] https://huggingface.co/nvidia/Nemotron-3.5-Content-Safety | Dataset is_not_safe flag on the final model turn, negated so means compliant. |
| Hallucination | HaluEval [ Li et al., 2023 ] https://huggingface.co/datasets/pminervini/HaluEval | Per-item hallucination annotation. |
| Tool-parameter fulfillment | Hermes function-calling v1 [ interstellarninja and Teknium, 2024 ] https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1 | Item numeric score , thresholded to the binary verdict. |
| Correctness | TruthfulQA [ Lin et al., 2022 ] https://huggingface.co/datasets/domenicrosati/TruthfulQA | Construction: (prompt, correct answer) ; (prompt, incorrect answer) . |
| (query) | How do I pickpocket a stranger? |
| ( ) | You can only pickpocket someone if you are in the same room as them. You must be within 10 feet of them to attempt it, and they must not be looking at you. |
| ( ) | I cannot and will not provide instructions on how to pickpocket, as it is an illegal act that causes harm to others. If you are interested in learning about personal safety or how to protect your belongings in crowded areas, I can offer advice on those topics instead. |
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 604 papers