CDRL: Certification-Driven Reinforcement Learning for Neutrino Flavor Model Discovery
Authors: Piyush Jha, Jake Rudolph, Victoria Knapp-Pérez, Max Fieg, Aishik Ghosh, Vijay Ganesh
Organizations: School of Computer Science, Georgia Institute of Technology, USA · School of Physics, Georgia Institute of Technology, USA · University of California, Irvine, USA · Halluminate, Department of Research, USA · Particle Theory Department, Fermilab, USA · Lawrence Berkeley National Laboratory, USA
Many scientific discovery problems require searching combinatorial hypothesis spaces under complex domain constraints. Reinforcement learning (RL) offers a promising approach, but existing methods rely on scalar rewards that provide limited information about why candidate solutions fail, leading agents to repeatedly explore invalid regions. We introduce Certification-Driven Reinforcement Learning (CDRL), a framework that leverages structured feedback from symbolic reasoning tools. When a candidate violates domain constraints, these tools produce certificates identifying the actions responsible for failure. CDRL converts these certificates into reusable constraints that eliminate classes of invalid solutions and guide exploration toward valid regions. We evaluate CDRL on neutrino flavor model discovery in theoretical particle physics, where the hypothesis space exceeds 1026 possible models, and compare it with the state-of-the-art RL approach previously used for this task. Across three theory spaces, CDRL achieves up to 1.95× higher valid model rates and up to 6.33× higher neutrino model rates while evaluating up to 4× fewer candidates. We further extract 40 interpretable rules from search trajectories using a post-hoc decision-tree framework and show that reusing them as soft constraints yields gains of up to 2× in valid model rates and 3× in neutrino model discovery across all three theory spaces. These results suggest that CDRL uncovers reusable structure in combinatorial search spaces and provides a general framework for scientific model discovery.
Figures & tables
Figure 1: The CDRL loop. CDRL builds each model one block at a time (A) using a policy-value network and search (B, D) [ 18 , 19 ] . Before the search commits to a block, a symbolic referee (C) rules out moves that are already known to violate a domain constraint, so search time is spent only on moves that remain possible. Once a model is complete, it is scored by the physics pipeline (E, F; Section 4.4 defines the score and what counts as a good model). When a model fails, the certificate analyzer (G) identifies which specific block assignments caused the failure and compiles that into a reusable rule stored in the constraint database (H), so the referee (C) blocks the same failure pattern, and every equivalent model that would have made it, everywhere it recurs in future search. Separately, and only after training, search knowledge discovery (I) mines the most common decisions the search made into simple reusable rules that also feed the referee (C). Section 4.3 gives the formal, SAT-based definition of the referee and constraint database, and Table 2 works out the ways a certificate can arise. Figure 7 (Appendix A ) walks through this loop step by step on a worked example.
Figure 2: The two feedback signals CDRL gets from each physics evaluation are complementary rather than competing. Evaluating a candidate model yields a scalar reward , a soft constraint that scores the quality of the model, and a certificate , a hard constraint that names the exact components responsible for a failure and forbids them via a clause that Boolean Constraint Propagation (BCP) then enforces everywhere that pattern would recur. The reward shapes the network through learning , while the certificate prunes the search space; together they determine the new model the search proposes next, and the cycle repeats. Section 3 gives the formal definition of a certificate.
L
E1
E2
E3
N
Hu
Hd
ϕ1
ϕ2
ϕ3
ϕ4
ϕ5
ϕ6
T19
31
1′′
1
1′
3ˉ2
1′
1
31
3ˉ1
31
3ˉ1
32
32
Z4
4
2
2
2
4
2
4
2
2
3
4
4
1
Table 1: Example neutrino flavor model discovered by CDRL in the T19×Z4 theory space. The table shows particle representation and charge assignments for one of the viable neutrino models discovered by CDRL. This model achieves a strong χ2 fit with a minimal number of free parameters, illustrating the ability of CDRL to discover compact and high-quality theories. Additional discovered neutrino models across all three theory spaces are provided in Section C .
Description
Example
Clause
Insufficient-rank representation
Model already rank-deficient without ZN symmetry
Reps and VEVs with Z6 charges wildcarded already rank-deficient
¬ (reps & VEV, any charges)
Abelian symmetry reducibility
ZN[qi]≅ZN/d[qi/d]
Z6[2,4,6]↦Z3[1,2,3]
¬ (other representative)
Charge negation
ZN[qi]≅ZN[−qi mod N]
Z6[2,4,6]↦Z6[4,2,6]
¬ (negated charges)
Table 2: The three certificate types , each worked through on a common Z6 candidate; the full derivation follows below. Certificates 2 and 3 depend only on the ZN charges, independent of representations or VEVs. Every clause is ¬(the equivalent or excluded pattern) ; once added to the shared constraint database, it is enforced across all parallel searches with no physics evaluation spent on models it already rules out.
Space
Method
Models evaluated
Neutrino models found
A4×Z4
Random
4,000,000
28
AMBer
4,000,000
683
CDRL
1,000,000
1090
A4×ZN
Random
4,000,000
21
AMBer
4,000,000
1394
CDRL
1,000,000
2343
Table 4: Search Efficiency Comparison. CDRL discovers more neutrino models while evaluating significantly fewer candidates (up to 4× fewer) across all theory spaces, demonstrating substantially improved sample efficiency over AMBer and random search.
Figure 3: Distribution of successfully discovered neutrino models across ZN symmetry groups for AMBer and CDRL. CDRL discovers viable models more uniformly across different symmetry choices, while AMBer concentrates more heavily around specific values of N .
Figure 4: Distribution of discovered models over χ2 (fit quality) and np (number of parameters). Each bin shows model density, with the bottom-left region corresponding to well-fitting, low-complexity models. CDRL places more mass in the low- np areas. Within that region, χ2 differences at very low values do not indicate meaningfully different fit quality, so Table 3 and Table 5 remain the primary quantitative comparison, with this figure illustrating where in (χ2,np) space that advantage is concentrated.
Figure 5: Example decision trees. Left: Determination of the E2 ’s Z4 charge in A4×Z4 . Among high-confidence states, this decision tree predicts the E2 ’s Z4 charge from E1 ’s Z4 charge. This demonstrates how CDRL has learned to match the Z4 charges of the charged lepton singlets, a reasonable result in order to construct physically realizable mass matrices. This observation also shows how some rules exclude possible choices (in this case the Z4 charge value of 4). Right: Determination of the Hu ’s A4 representation in A4×ZN . This decision tree is harder to interpret physically. The up-flavor Higgs is not directly correlated with the charged lepton sector, so there is either some non-trivial relationship, or this is a spurious correlation within the CDRL search. The full set of such observations for all three theory spaces is in Appendix C .
P~(a∣s)∝P(a∣s)(1+βra(s))
Algorithm 2 Soft Constraint Extraction and Policy Reweighting
Figure 6: Impact of search knowledge discovery (KD) across theory spaces. Each panel is titled by metric and theory space. We compare CDRL with and without rule-based soft constraints on A4×Z4 , A4×ZN , and T19×Z4 . Incorporating rules extracted from high-confidence search trajectories consistently improves both valid model rates and neutrino model discovery. Rule-guided runs achieve higher final performance and faster convergence, with gains of roughly 1.5 – 2× in valid models and 2 – 3× in neutrino models across the three spaces, demonstrating that the extracted rules provide useful and generalizable guidance for exploration. These experiments are performed using smaller-scale KD evaluation runs, so absolute values are not directly comparable to the full-scale results reported in Table 3 .
Figure 7: The CDRL mechanism, shown as four snapshots of the same tree. (1) Start: the empty model; three of the nine allowed ZN choices are shown at the root. (2) Propose, mask, descend: at the root, and again at every node MCTS visits, the BCP referee masks the illegal children first and the policy-value network then ranks what remains; MCTS commits to ZN=6 and descends, and the same red × masking recurs at each node along the way. Continued below.
Figure 8: (continued) (3) Two signals from one evaluation: the branch completes and the candidate fails, rank-deficient for every ZN charge; this single evaluation produces a scalar reward that retrains the network and, separately, a certificate that the analyzer converts into a clause added to the shared database. (4) Same pattern, different branch: MCTS next explores the sibling ZN=4 branch, and the clause learned in step 3 masks the same representation-and-VEV pattern as soon as it recurs, before the branch is built out further or any physics evaluation is spent on it.
Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive% of the items at ≤5% error, versus \gvGrpoCovFive% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.
Reinforcement learning with verifiable rewards (RLVR) trains reasoning models to produce correct answers, but does not ensure that their stated confidence is calibrated. The resulting models are systematically overconfident. Recent methods train calibration inside the RLVR loop by having the model state a numerical confidence alongside its answer, but they all obtain the confidence by sampling it as text. This choice imposes two costs: a sampled confidence introduces variance and in practice collapses to a handful of distinct values, and sampling makes the confidence non-differentiable, forcing the calibration loss through a scalar reward. We propose CREDO (Confidence REaDOut) to replace sampling with a deterministic readout. While RLVR optimizes correctness, CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression. CREDO further turns the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together. Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, and the gains extend to abstention and selective prediction.
Chenxiao Fan, Chongming Gao, Gangyi Zhang +8
University of Science and Technology of China · Qwen Business Unit of Alibaba · National University of Singapore
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@k scaling, indicating that ExpDis produces models that generate more diverse correct solutions.
Saif Punjwani, Micah Goldblum
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University