Cross-Question Reliable Reinforcement Learning
Organizations: TU Darmstadt & hessian.AI, Germany
Abstract
AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair individually, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.
Figures & tables
| ID | OOD | ||||||||||
| Model | Init | C@0 | C@1 | C@5 | AURC | Acc | C@0 | C@1 | C@5 | AURC | Acc |
| RLVR-Classifier [ 15 ] | N/A | 4.6 | 5.9 | 33.6 | 12.8 | 73.7 | 0.5 | 0.5 | 1.5 | 45.2 | 52.5 |
| CAR-WeightedBrier [ 57 ] | Cold | 0.0 | 0.0 | 0.0 | 20.6 | 71.8 | 0.1 | 0.1 | 0.1 | 41.7 | 51.3 |
| CAR-MSCR [ 57 ] | Cold | 0.0 | 0.0 | 0.0 | 19.2 | 72.0 | 0.1 | 0.1 | 7.0 | 37.7 | 56.7 |
| BC-Beta [ 54 ] | Cold | 0.0 | 0.0 | 0.0 | 17.6 | 71.8 | 0.0 | 0.0 | 11.6 | 37.6 | 55.7 |
| RLSR [ 36 ] | Cold | 0.0 | 0.0 | 0.0 | 21.7 | 72.7 | 0.0 | 0.0 | 0.0 | 51.0 | 43.9 |
| ID | OOD | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | C@0 | C@1 | C@5 | AURC | Acc | C@0 | C@1 | C@5 | AURC | Acc |
| RLVR-Classifier [ 15 ] | 1.6 | 1.6 | 14.2 | 6.5 | 87.7 | 0.4 | 0.4 | 6.0 | 42.0 | 53.9 |
| CAR-WeightedBrier [ 57 ] | 0.0 | 0.0 | 0.0 | 6.4 | 85.8 | 0.0 | 0.0 | 16.5 | 35.6 | 57.3 |
| CAR-MSCR [ 57 ] | 0.0 | 0.0 | 0.0 | 7.2 | 87.0 | 0.0 | 7.4 | 14.8 | 36.2 | 56.0 |
| BC-Beta [ 54 ] | 0.1 | 0.1 | 0.1 | 7.2 | 81.9 | 0.9 | 5.6 | 14.8 | 30.5 | 63.4 |
| RLSR [ 36 ] | 0.0 | 0.0 | 0.0 | 8.6 | 86.6 | 0.0 | 0.0 | 14.5 | 37.7 | 58.8 |
| ID | OOD | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Train Data | Method | C@0 | C@1 | C@5 | C@10 | AURC | Acc | C@0 | C@1 | C@5 | C@10 | AURC | Acc |
| Math | RLCR [ 15 ] | 0.0 | 0.0 | 0.0 | 0.0 | 18.2 | 73.9 | 0.0 | 0.0 | 4.9 | 33.1 | 38.0 | 54.0 |
| CARE (Ours) | 0.3 | 0.3 | 0.7 | 37.7 | 15.5 | 74.4 | 0.8 | 0.8 | 8.8 | 26.2 | 36.5 | 54.5 | |
| HotpotQA | RLCR [ 15 ] | 0.0 | 0.0 | 0.0 | 92.3 | 6.8 | 86.7 | 0.0 | 0.0 | 7.8 | 14.1 | 36.1 | 55.7 |
| CARE (Ours) | 0.0 | 0.0 | 58.2 | 93.6 | 6.1 | 88.0 | 1.4 | 1.9 | 5.0 | 18.5 | 33.9 | 56.5 | |
| Reward terms | ID | OOD | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | C@0 | C@1 | C@5 | C@10 | AURC | Acc | C@0 | C@1 | C@5 | C@10 | AURC | Acc | |||||
| 1 | ✓ | ✓ | ✓ | ✓ | ✓ | 5.4 | 18.2 | 42.4 | 52.9 | 12.5 | 73.1 | 2.9 | 5.4 | 15.9 | 26.4 | 37.7 | 53.6 |
| 2 | ✓ | ✓ | ✓ | ✓ | 3.0 | 3.0 | 40.3 | 55.8 | 12.1 | 73.9 | 2.6 | 4.0 | 16.1 | 22.1 | 36.9 | 56.5 | |
| 3 | ✓ | ✓ | ✓ | ✓ | 2.2 | 3.0 | 33.1 | 54.2 | 12.7 | 72.3 | 0.5 | 2.5 | 14.4 | 28.7 | 36.8 | 53.7 | |
| 4 | ✓ | ✓ | ✓ | 0.6 | 0.6 | 16.3 | 33.3 | 19.3 | 72.4 | 0.4 | 0.4 | 4.6 | 20.9 | 41.5 | 55.7 | ||
| 5 | ✓ | ✓ | ✓ | ✓ | 3.6 | 3.6 | 30.2 | 51.6 | 13.2 | 72.3 | 2.0 | 6.9 | 18.5 | 26.3 | 33.5 | 59.7 | |
| ID Math | OOD General Reasoning and Knowledge | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Scope | Size | C@0 | C@1 | C@5 | C@10 | AURC | Acc | C@0 | C@1 | C@5 | C@10 | AURC | Acc |
| Per-group | 32 | 0.0 | 0.0 | 0.0 | 33.3 | 19.9 | 72.6 | 0.0 | 0.0 | 0.0 | 3.8 | 46.3 | 46.2 |
| Per-worker | 384 | 5.4 | 18.2 | 42.4 | 52.9 | 12.5 | 73.1 | 2.9 | 5.4 | 15.9 | 26.4 | 37.7 | 53.6 |
| Across-workers | 2,304 | 2.2 | 2.2 | 28.7 | 54.3 | 13.3 | 71.9 | 1.5 | 1.5 | 18.2 | 31.0 | 34.3 | 57.5 |
| Rolling buffer | 5,000 | 0.0 | 0.0 | 0.0 | 33.3 | 24.3 | 73.7 | 0.0 | 0.0 | 0.0 | 0.0 | 42.0 | 57.7 |
| Rolling buffer | 10,000 | 0.0 | 0.0 | 0.0 | 33.3 | 33.9 | 73.3 | 0.3 | 0.3 | 0.8 | 20.1 | 41.0 | 59.6 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Topic | Example |
| Training Datasets | ||
| Big-Math-Digits | Mixed-domain math | Q: Given and , find . A: 10 |
| HotpotQA | Wikipedia multi-hop QA | Q: Were Scott Derrickson and Ed Wood of the same nationality? A: yes |
| Test Benchmarks | ||
| MATH-500 | Competition math | Q: Convert from rectangular to polar coordinates. A: |
| GSM8K | Grade-school math | Q: Janet’s ducks lay 16 eggs; after eating 3 and baking with 4, how much money does she make selling the rest at $2 each? A: 18 |
| Model | Init | C@0 | C@1 | C@5 | C@10 | C@15 | C@20 | C@25 | C@30 | AURC | Acc |
|---|---|---|---|---|---|---|---|---|---|---|---|
| RLVR-Classifier [ 15 ] | N/A | 8.2 | 12.1 | 25.5 | 49.0 | 65.1 | 80.1 | 89.2 | 99.2 | 11.9 | 69.8 |
| CAR-WeightedBrier [ 57 ] | Cold | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 81.4 | 85.3 | 100.0 | 18.5 | 71.3 |
| CAR-MSCR [ 57 ] | Cold | 0.1 | 0.1 | 0.1 | 0.1 | 69.5 | 80.4 | 87.8 | 87.8 | 17.5 | 68.8 |
| BC-Beta [ 54 ] | Cold | 0.0 | 0.0 | 0.0 | 51.9 | 71.6 | 81.5 | 86.6 | 100.0 | 13.9 | 70.0 |
| RLSR [ 36 ] | Cold | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 83.7 | 85.6 | 100.0 | 20.0 | 70.7 |
| ORCE-SC [ 34 ] | Cold | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 100.0 | 26.2 | 71.6 |
| Model | Init | C@0 | C@1 | C@5 | C@10 | C@15 | C@20 | C@25 | C@30 | AURC | Acc |
|---|---|---|---|---|---|---|---|---|---|---|---|
| RLVR-Classifier [ 15 ] | N/A | 0.5 | 0.5 | 1.5 | 19.7 | 20.4 | 20.4 | 20.7 | 21.6 | 45.2 | 52.5 |
| CAR-WeightedBrier [ 57 ] | Cold | 0.1 | 0.1 | 0.1 | 0.1 | 15.0 | 32.5 | 39.9 | 55.0 | 41.7 | 51.3 |
| CAR-MSCR [ 57 ] | Cold | 0.1 | 0.1 | 7.0 | 19.1 | 36.1 | 36.5 | 40.1 | 40.1 | 37.7 | 56.7 |
| BC-Beta [ 54 ] | Cold | 0.0 | 0.0 | 11.6 | 19.6 | 32.3 | 36.1 | 39.8 | 40.3 | 37.6 | 55.7 |
| RLSR [ 36 ] | Cold | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 13.0 | 13.0 | 51.0 | 43.9 |
| ORCE-SC [ 34 ] | Cold | 0.0 | 0.0 | 0.0 | 20.0 | 20.0 | 20.0 | 38.1 | 40.0 | 40.9 | 56.8 |
| Model | Init | C@0 | C@1 | C@5 | C@10 | C@15 | C@20 | C@25 | C@30 | AURC | Acc |
|---|---|---|---|---|---|---|---|---|---|---|---|
| RLVR-Classifier [ 15 ] | N/A | 4.6 | 5.9 | 33.6 | 52.7 | 59.2 | 75.4 | 85.5 | 92.7 | 12.8 | 73.7 |
| CAR-WeightedBrier [ 57 ] | Cold | 0.0 | 0.0 | 0.0 | 33.3 | 33.3 | 60.5 | 61.8 | 66.7 | 20.6 | 71.8 |
| CAR-MSCR [ 57 ] | Cold | 0.0 | 0.0 | 0.0 | 33.4 | 56.5 | 60.1 | 62.6 | 62.6 | 19.2 | 72.0 |
| BC-Beta [ 54 ] | Cold | 0.0 | 0.0 | 0.0 | 50.6 | 57.2 | 60.5 | 62.2 | 66.7 | 17.6 | 71.8 |
| RLSR [ 36 ] | Cold | 0.0 | 0.0 | 0.0 | 33.3 | 33.3 | 61.2 | 61.9 | 66.7 | 21.7 | 72.7 |
| ORCE-SC [ 34 ] | Cold | 0.0 | 0.0 | 0.0 | 33.3 | 33.3 | 33.3 | 33.3 | 66.7 | 23.6 | 75.0 |
| Model | C@0 | C@1 | C@5 | C@10 | C@15 | C@20 | C@25 | C@30 | AURC | Acc |
|---|---|---|---|---|---|---|---|---|---|---|
| RLVR-Classifier [ 15 ] | 1.6 | 1.6 | 14.2 | 92.8 | 100.0 | 100.0 | 100.0 | 100.0 | 6.5 | 87.7 |
| CAR-WeightedBrier [ 57 ] | 0.0 | 0.0 | 0.0 | 90.0 | 100.0 | 100.0 | 100.0 | 100.0 | 6.4 | 85.8 |
| CAR-MSCR [ 57 ] | 0.0 | 0.0 | 0.0 | 89.6 | 100.0 | 100.0 | 100.0 | 100.0 | 7.2 | 87.0 |
| BC-Beta [ 54 ] | 0.1 | 0.1 | 0.1 | 89.7 | 89.7 | 100.0 | 100.0 | 100.0 | 7.2 | 81.9 |
| RLSR [ 36 ] | 0.0 | 0.0 | 0.0 | 85.6 | 100.0 | 100.0 | 100.0 | 100.0 | 8.6 | 86.6 |
| ORCE-SC [ 34 ] | 0.0 | 0.0 | 0.0 | 0.0 | 100.0 | 100.0 | 100.0 | 100.0 | 10.7 | 85.6 |
| Model | C@0 | C@1 | C@5 | C@10 | C@15 | C@20 | C@25 | C@30 | AURC | Acc |
|---|---|---|---|---|---|---|---|---|---|---|
| RLVR-Classifier [ 15 ] | 0.4 | 0.4 | 6.0 | 17.5 | 17.9 | 33.7 | 34.7 | 36.9 | 42.0 | 53.9 |
| CAR-WeightedBrier [ 57 ] | 0.0 | 0.0 | 16.5 | 32.4 | 33.4 | 44.0 | 46.0 | 46.0 | 35.6 | 57.3 |
| CAR-MSCR [ 57 ] | 0.0 | 7.4 | 14.8 | 31.4 | 32.2 | 42.7 | 44.3 | 44.3 | 36.2 | 56.0 |
| BC-Beta [ 54 ] | 0.9 | 5.6 | 14.8 | 34.0 | 40.7 | 42.5 | 50.0 | 50.0 | 30.5 | 63.4 |
| RLSR [ 36 ] | 0.0 | 0.0 | 14.5 | 32.8 | 33.3 | 33.3 | 33.3 | 45.6 | 37.7 | 58.8 |
| ORCE-SC [ 34 ] | 0.0 | 0.0 | 14.9 | 16.7 | 31.7 | 32.5 | 43.6 | 45.4 | 38.5 | 55.2 |
| ID | OOD | ||||||
| Training | Method | AUROC | Brier | ECE | AUROC | Brier | ECE |
| RLVR-Classifier [ 15 ] | 78.2 | .149 | 9.3 | 54.6 | .343 | 34.3 | |
| RLVR [ 15 ] | 46.5 | .209 | 12.3 | 51.6 | .283 | 25.7 | |
| Math | RLCR [ 15 ] | 67.8 | .166 | 8.7 | 61.5 | .278 | 25.7 |
| CARE | 83.7 | .148 | 12.5 | 67.3 | .233 | 22.1 | |
| RLVR-Classifier [ 15 ] | 72.1 | .102 | 5.8 | 58.3 | .320 | 30.1 | |
| ID | OOD | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Reward transform | C@0 | C@1 | C@5 | AURC | Acc | C@0 | C@1 | C@5 | AURC | Acc |
| Linear ( ) | 0.6 | 0.6 | 36.1 | 13.7 | 72.7 | 1.2 | 4.3 | 17.1 | 34.9 | 58.8 |
| Decay exponent ( ) | 3.4 | 9.7 | 35.3 | 13.5 | 72.3 | 0.4 | 0.4 | 16.3 | 35.8 | 56.3 |
| Squared ( ) | 5.4 | 18.2 | 42.4 | 12.5 | 73.1 | 2.9 | 5.4 | 15.9 | 37.7 | 53.6 |
LLM Abstention
Large language model abstention studies when AI systems should answer, ask for more information, defer, or withhold unsupported claims.
See all 111 papers