Anchor and Check: Deterministic Cross-Prompt, Cross-Model Reconciliation for Auditable LLM Decisions
Abstract
Pipelines that let a large language model (LLM) make discrete decisions need to know which decisions they may act on unattended, and a record that explains each one. We study a small, fully specified reconciliation contract: commit the target model's greedy answer (the anchor), draw one or two greedy checks that reach the answer by a different route (an alternate prompt, or a model from another family), and let a pure function return one of five outcomes, of which only Agree is acted on. On 48,000 model-item evaluations over 6,000 questions, with every method given access to verbalized confidence as a tie-breaker, the main finding concerns where calls come from: pooling another family's samples raises semantic entropy's AUROC by 1.7 to 4.0 points at every budget, and a same-model temperature check is the weakest design. As complete pipelines, the point estimates of the three-call contract and pooled semantic entropy over 3 or 5 samples differ by less than one point, with intervals containing zero. As detectors of the anchor's errors, the contract is ahead of pooled entropy with 3 and 5 samples under both tie-breaker conventions; with 10 samples the point estimates differ by less than one point and the sign depends on the convention. Under the strict Agree gate it automates 70% of items and still acts on 16% of errors. What the contract adds is a fixed committed answer, an explicit outcome and a decision replayable from its record; generation itself was not reproducible on the hosted API.
Figures & tables
| Key type | Normalization | Equality |
|---|---|---|
| Numeric | parse; strip units and thousands separators | numeric equality within tolerance |
| Categorical | case-fold, strip whitespace | exact |
| Short text | case-fold, strip punctuation and articles | exact |
| Structured | canonical JSON (sorted keys, normalized scalars) | exact |
| Date | parse ISO or prose; missing month/day is a wildcard | component-wise match |
| Method | Calls | Accuracy | AUROC | AUROC (native) | USD / 1k items |
|---|---|---|---|---|---|
| Greedy + verbalized confidence | 1 | 0.764 | 0.663 | 0.515 | 0.070 |
| Anchor + creative (ablation) | 2 | 0.764 | 0.754 | 0.679 | 0.141 |
| Anchor + alt | 2 | 0.764 | 0.787 | 0.728 | 0.138 |
| Anchor + cheap | 2 | 0.764 | 0.802 | 0.761 | 0.090 |
| Anchor + peer | 2 | 0.764 | 0.819 | 0.776 | 0.141 |
| Anchor + alt + cheap | 3 | 0.764 | 0.846 | 0.822 | 0.158 |
| Method | Calls | USD / 1k | Accuracy | AUROC | Contract baseline |
| Anchor + peer (contract) | 2 | 0.141 | 0.764 | 0.819 | |
| Pooled greedy SE-2 (anchor, peer) | 2 | 0.141 | 0.765 | 0.828 | |
| Pooled SE-2 ( , 1 target + 1 peer) | 2 | 0.142 | 0.752 | 0.842 | |
| Anchor + alt + peer (contract) | 3 | 0.208 | 0.764 | 0.852 | |
| Pooled greedy SE-3 (anchor, alt, peer) | 3 | 0.208 | 0.779 | 0.838 | |
| Pooled SE-3 (2 target + 1 peer) | 3 | 0.212 | 0.767 | 0.849 |
| Method | Total calls | AUROC | Contract baseline |
|---|---|---|---|
| Anchor + peer (contract) | 2 | 0.819 | |
| Pooled greedy SE-2 (anchor, peer) | 2 | 0.831 | |
| Pooled SE-2 ( , 1 target + 1 peer) | 3 | 0.785 | |
| Anchor + alt + peer (contract) | 3 | 0.852 | |
| Pooled greedy SE-3 (anchor, alt, peer) | 3 | 0.854 | |
| Pooled SE-3 (2 target + 1 peer) | 4 | 0.819 |
| Method | Extra | Total | AUROC | Diff. |
|---|---|---|---|---|
| Anchor + peer (reference) | 1 | 2 | 0.820 | |
| Pooled greedy SE-2 (anchor, peer) | 1 | 2 | 0.820 | (identical ranking) |
| Same-model, agreement: anchor + 1 sample | 1 | 2 | 0.756 | |
| SE-2 (single model) | 2 | 3 | 0.746 | |
| Pooled SE-2 (1 target + 1 peer) | 2 | 3 | 0.784 | |
| SE-5 (single model) | 5 | 6 | 0.802 |
| Anchor | Alternate prompt | Peer | Temperature 0.8 (same model) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Task | Acc. | Acc. | Same wrong | Agree if right | Acc. | Same wrong | Agree if right | Acc. | Same wrong | Agree if right |
| GSM8K | 0.946 | 0.899 | 0.43 | 0.93 | 0.946 | 0.32 | 0.97 | 0.927 | 0.46 | 0.96 |
| SVAMP | 0.943 | 0.935 | 0.39 | 0.97 | 0.943 | 0.30 | 0.97 | 0.936 | 0.43 | 0.98 |
| MMLU | 0.805 | 0.788 | 0.54 | 0.91 | 0.805 | 0.44 | 0.90 | 0.793 | 0.72 | 0.95 |
| ARC | 0.941 | 0.941 | 0.49 | 0.97 | 0.941 | 0.41 | 0.97 | 0.932 | 0.77 | 0.98 |
| TriviaQA | 0.686 | 0.658 | 0.37 | 0.86 | 0.686 | 0.22 | 0.83 | 0.664 | 0.46 | 0.91 |
| Held-out test half | CommonsenseQA | |||
|---|---|---|---|---|
| Method | AUROC | vs SE | AUROC | vs SE |
| Greedy + verbalized confidence | 0.663 | 0.615 | ||
| Anchor + creative (ablation) | 0.750 | 0.660 | ||
| Anchor + alt | 0.783 | 0.738 | ||
| Anchor + peer | 0.811 | SE-5: | 0.752 | SE-5: |
| Anchor + alt + peer | 0.846 | SE-10: | 0.800 | SE-10: |
| Strict gate (own rule) | Score threshold at 84.0% automation | |||||
| Method | Automated | Error (auto.) | Caught | Error (auto.) | Caught | Precision |
| Greedy + verbalized confidence | – | – | – | 19.0% | 32.5% | 48.0% |
| Anchor + creative (ablation) | 84.0% | 13.6% | 51.7% | 13.6% | 51.7% | 76.3% |
| Anchor + peer | 76.2% | 7.7% | 75.2% | 13.3% | 52.7% | 77.9% |
| Anchor + alt + peer | 70.1% | 5.4% | 84.0% | 11.6% | 58.6% | 86.5% |
| Semantic entropy, 5 samples | 72.5% | 7.9% | 75.2% | 12.3% | 55.2% | 79.6% |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | creative | alt | peer | alt + peer | 4-call | SE-3 | SE-5 | SE-10 | Pooled SE-5 |
|---|---|---|---|---|---|---|---|---|---|
| Ministral 8B | 0.729 | 0.759 | 0.806 | 0.854 | 0.877 | 0.830 | 0.845 | 0.843 | 0.870 |
| Mistral Large 3 | 0.773 | 0.804 | 0.823 | 0.863 | 0.873 | 0.795 | 0.812 | 0.805 | 0.846 |
| Qwen3 32B | 0.734 | 0.787 | 0.816 | 0.852 | 0.867 | 0.794 | 0.815 | 0.830 | 0.852 |
| Qwen3 235B A22B | 0.707 | 0.768 | 0.809 | 0.837 | 0.853 | 0.768 | 0.785 | 0.796 | 0.843 |
| gpt-oss-20b | 0.804 | 0.802 | 0.857 | 0.871 | 0.889 | 0.842 | 0.856 | 0.850 | 0.846 |
| gpt-oss-120b | 0.798 | 0.816 | 0.849 | 0.859 | 0.871 | 0.824 | 0.826 | 0.838 | 0.850 |
| Pair | Anchor | Accuracy | AUROC (peer) | AUROC (alt + peer) | USD / 1k items |
|---|---|---|---|---|---|
| Ministral 8B + GLM 4.7-flash | Ministral 8B | 0.706 | 0.806 | 0.854 | 0.040 |
| GLM 4.7-flash | 0.730 | 0.802 | 0.856 | 0.040 | |
| Qwen3 32B + gpt-oss-20b | Qwen3 32B | 0.746 | 0.816 | 0.852 | 0.072 |
| gpt-oss-20b | 0.730 | 0.857 | 0.871 | 0.072 | |
| Mistral Large 3 + GLM 5 | Mistral Large 3 | 0.805 | 0.823 | 0.863 | 0.331 |
| GLM 5 | 0.817 | 0.788 | 0.826 | 0.331 |
| Short-text tasks | Full grid | |||
|---|---|---|---|---|
| Method | Exact | Entail | Exact | Entail |
| Greedy + verbalized confidence | 0.662 | 0.662 | 0.663 | 0.663 |
| Anchor + creative (ablation) | 0.752 | 0.741 | 0.754 | 0.750 |
| Anchor + alt | 0.782 | 0.769 | 0.787 | 0.783 |
| Anchor + peer | 0.802 | 0.787 | 0.819 | 0.814 |
| Anchor + alt + peer | 0.837 | 0.820 | 0.852 | 0.847 |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 866 papers