Answer-only reinforcement learning (RL) trains reasoning models to solve fully specified problems, but many realistic queries omit a premise needed for a unique answer. In this setting, the useful response is not always refusal: the model should ask for the missing premise, condition its answer on the unknown quantity, or abstain when no informative conditional response is available. We present Ask-Condition-Abstain Reinforcement Learning (ACA-RL), a data-augmented RL framework for this setting. Its reasoning-graph-guided pipeline converts well-posed problems into missing-premise training instances with localized gap annotations; ACA-RL then trains on these instances with a structured reward over five observable response behaviors. We also introduce the Missing-Premise Benchmark (MPB), a 274-instance human-verified benchmark spanning mathematical, logical, and real-world word problems. Across Qwen3 and Llama models, ACA-RL consistently improves on MPB while preserving competitive performance on well-posed reasoning tasks. Together with the released code, MPB, and training data, this work supports a new mission for NLP evaluation: measuring whether models can recognize when a task is underdetermined and handle uncertainty, not only whether they can answer fully specified questions.
Figures & tables
Figure 1: Motivation and framework overview. Answer-only RL trains reasoning models on fully specified questions with verifiable answers, but real user queries often omit critical premises. ACA-RL augments reasoning data with missing-premise variants and uses a structured behavioral reward to prefer asking, conditioning, or abstaining. MPB measures whether responses merely refuse or expose the missing premise in a useful form.
Figure 2: Reasoning-graph-guided missing-premise synthesis pipeline. ( 1 ) Construct a directed acyclic reasoning graph from a well-posed problem; ( 2 ) trace the solution path to identify critical constraints; ( 3 ) perturb key conditions to create a logically underspecified variant; ( 4 ) rewrite and verify the instance to ensure plausibility and unanswerability. The pipeline generates controlled synthetic data for training missing-premise response behavior.
Method
Missing-Premise
Unanswerable
Mathematics
MPB
UMWP
SUM
GSM8K
MATH-500
AIME’24
Qwen3-8B (Reasoning Model)
Cold-start SFT
22.81
33.53
20.51
83.69
75.20
37.08
Vanilla PPO
8.66
8.84
9.41
93.85
93.40
64.58
IDK-RL
48.72
45.74
42.95
88.02 (-5.83)
92.40 (-1.00)
63.33 (-1.25)
ACA-RL (Ours)
51.73
51.50
46.91
91.50 (-2.35)
92.60 (-0.80)
64.16 (-0.42)
Table 1: Performance across missing-premise and general reasoning benchmarks for different model architectures and scales. Robustness is measured by MPB and by the same ask/condition/abstain Behavior Score on two unanswerable benchmarks ( UMWP , SUM ); general reasoning is measured by Pass@1 on GSM8K , MATH-500 , and AIME’24 (averaged over 8 runs). Red values indicate the performance change relative to the strongest RL baseline within each model group.
Method
LiveBench Avg.
SciBench Overall
Cold-start SFT
51.9
47.0
Vanilla PPO/RLVR
59.0
56.0
ACA-RL (Ours)
59.0
53.7
Table 2: General capability check on well-posed benchmarks for Qwen3-8B. ACA-RL preserves the LiveBench average and remains competitive on SciBench, indicating that missing-premise training does not collapse into over-refusal on ordinary tasks.
Source
MPB
GSM8K
AIME’24
Treecut
15.41
94.01
67.50
SUM
47.35
91.05
59.58
Ours
51.73
91.50
64.16
Table 3: Data-source ablation under the same data budget. The reasoning-graph-guided synthesis gives the highest MPB Behavior Score while maintaining competitive GSM8K and AIME’24 performance.
Portion
MPB
GSM8K
AIME’24
50%
53.19
82.10
52.56
30%
43.79
90.14
60.83
10%
28.28
92.49
63.33
Table 4: Training on 10K samples over 100 steps shows the trade-off between missing-premise robustness and standard reasoning as the missing-premise ratio changes. We use 30% as the default because it preserves substantially more general reasoning performance than 50% while improving MPB over 10%.
Method
UMWP
SUM
Behavior Score
IDK Score
Behavior Score
IDK Score
Cold-start SFT
33.53
72.57
20.51
45.77
Vanilla PPO
8.84
74.11
9.41
52.11
IDK-RL
45.74
95.23
42.95
91.19
ACA-RL
51.50
91.30
46.91
84.85
Table 5: Comparison between Behavior Score and traditional IDK score. IDK score measures conservative abstention, while Behavior Score additionally rewards responses that use available information or ask for missing premises. ACA-RL improves Behavior Score while retaining competitive IDK behavior.
Figure 3: Effect of ACA-RL training steps. Training on our synthetic missing-premise data increases MPB scores in this run while keeping the reported general benchmark scores relatively stable.
Figure 4: As ACA-RL training progresses, abstention emerges rapidly in early stages, while conditional formulation and elicitation increase more gradually.
Figure 5: Impact of cold-start SFT on ACA-RL convergence. SFT initialization improves learning efficiency and final MPB score under the plotted training budget.
Method
MPB
UMWP
SUM
Cold-SFT
22.81
33.53
20.51
Vanilla PPO
8.66
8.84
9.41
LM Introspection
9.58
32.01
13.99
ACA-RL (Ours)
51.73
51.50
46.91
Table 6: Comparison with training-free verbalized uncertainty prompting on Qwen3-8B.
Figure 6: Effect of training data size on ACA-RL performance. Larger generated datasets lead to higher final scores across the plotted training steps on GSM8K and MPB.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Reward Score
Behavior Score
Description
-1.0
0
Silent Hallucination
-0.3
25
Explicit Assumption
0.3
50
Abstains from Answering
0.6
75
Conditions on a Variable
1.0
100
Asks for the Missing Premise
Appendix
Table 9: Mapping between ask/condition/abstain behavior labels and final Behavior Scores.
Category
Hyperparameter
Value
Trainer
Nodes
4
GPUs per node
8
Total steps
400
Gradient checkpointing
True
Algorithm
Advantage estimator
GAE( λ =1, γ =1)
Use KL in reward
False
Appendix
Table 10: Hyperparameters of the PPO algorithm implemented based on the veRL framework.