CDRL: Certification-Driven Reinforcement Learning for Neutrino Flavor Model Discovery

Piyush Jha; Jake Rudolph; Victoria Knapp-Pérez; Max Fieg; Aishik Ghosh; Vijay Ganesh

cs.AIAug 21, 2026

CDRL: Certification-Driven Reinforcement Learning for Neutrino Flavor Model Discovery

Authors: Piyush Jha, Jake Rudolph, Victoria Knapp-Pérez, Max Fieg, Aishik Ghosh, Vijay Ganesh

Organizations: School of Computer Science, Georgia Institute of Technology, USA · School of Physics, Georgia Institute of Technology, USA · University of California, Irvine, USA · Halluminate, Department of Research, USA · Particle Theory Department, Fermilab, USA · Lawrence Berkeley National Laboratory, USA

Abstract

Many scientific discovery problems require searching combinatorial hypothesis spaces under complex domain constraints. Reinforcement learning (RL) offers a promising approach, but existing methods rely on scalar rewards that provide limited information about why candidate solutions fail, leading agents to repeatedly explore invalid regions. We introduce Certification-Driven Reinforcement Learning (CDRL), a framework that leverages structured feedback from symbolic reasoning tools. When a candidate violates domain constraints, these tools produce certificates identifying the actions responsible for failure. CDRL converts these certificates into reusable constraints that eliminate classes of invalid solutions and guide exploration toward valid regions. We evaluate CDRL on neutrino flavor model discovery in theoretical particle physics, where the hypothesis space exceeds $10^{26}$ possible models, and compare it with the state-of-the-art RL approach previously used for this task. Across three theory spaces, CDRL achieves up to 1.95 $\times$ higher valid model rates and up to 6.33 $\times$ higher neutrino model rates while evaluating up to 4 $\times$ fewer candidates. We further extract 40 interpretable rules from search trajectories using a post-hoc decision-tree framework and show that reusing them as soft constraints yields gains of up to 2 $\times$ in valid model rates and 3 $\times$ in neutrino model discovery across all three theory spaces. These results suggest that CDRL uncovers reusable structure in combinatorial search spaces and provides a general framework for scientific model discovery.

Figures & tables

Figure 1: The CDRL loop. CDRL builds each model one block at a time (A) using a policy-value network and search (B, D) [ 18 , 19 ] . Before the search commits to a block, a symbolic referee (C) rules out moves that are already known to violate a domain constraint, so search time is spent only on moves that remain possible. Once a model is complete, it is scored by the physics pipeline (E, F; Section 4.4 defines the score and what counts as a good model). When a model fails, the certificate analyzer (G) identifies which specific block assignments caused the failure and compiles that into a reusable rule stored in the constraint database (H), so the referee (C) blocks the same failure pattern, and every equivalent model that would have made it, everywhere it recurs in future search. Separately, and only after training, search knowledge discovery (I) mines the most common decisions the search made into simple reusable rules that also feed the referee (C). Section 4.3 gives the formal, SAT-based definition of the referee and constraint database, and Table 2 works out the ways a certificate can arise. Figure 7 (Appendix A ) walks through this loop step by step on a worked example.

Figure 2: The two feedback signals CDRL gets from each physics evaluation are complementary rather than competing. Evaluating a candidate model yields a scalar reward , a soft constraint that scores the quality of the model, and a certificate , a hard constraint that names the exact components responsible for a failure and forbids them via a clause that Boolean Constraint Propagation (BCP) then enforces everywhere that pattern would recur. The reward shapes the network through learning , while the certificate prunes the search space; together they determine the new model the search proposes next, and the cycle repeats. Section 3 gives the formal definition of a certificate.

L

E_{1}

E_{2}

E_{3}

N

H_{u}

H_{d}

\phi_{1}

\phi_{2}

\phi_{3}

\phi_{4}

\phi_{5}

\phi_{6}

T_{19}

\mathbf{3}_{1}

\mathbf{1}^{\prime\prime}

\mathbf{1}

\mathbf{1}^{\prime}

\bar{\mathbf{3}}_{2}

\mathbf{1}^{\prime}

\mathbf{1}

\mathbf{3}_{1}

\bar{\mathbf{3}}_{1}

\mathbf{3}_{1}

\bar{\mathbf{3}}_{1}

\mathbf{3}_{2}

\mathbf{3}_{2}

\mathbb{Z}_{4}

4

2

2

2

4

2

4

2

2

3

4

4

1

Table 1: Example neutrino flavor model discovered by CDRL in the

T_{19}\times\mathbb{Z}_{4}

theory space. The table shows particle representation and charge assignments for one of the viable neutrino models discovered by CDRL. This model achieves a strong

\chi^{2}

fit with a minimal number of free parameters, illustrating the ability of CDRL to discover compact and high-quality theories. Additional discovered neutrino models across all three theory spaces are provided in Section C .

Description	Example	Clause
Insufficient-rank representation
Model already rank-deficient without $\mathbb{Z}_{N}$ symmetry	Reps and VEVs with $\mathbb{Z}_{6}$ charges wildcarded already rank-deficient	$\neg$ (reps & VEV, any charges)
Abelian symmetry reducibility
$\mathbb{Z}_{N}[q_{i}]\cong\mathbb{Z}_{N/d}[q_{i}/d]$	$\mathbb{Z}_{6}\,[2,4,6]\mapsto\mathbb{Z}_{3}\,[1,2,3]$	$\neg$ (other representative)
Charge negation
$\mathbb{Z}_{N}[q_{i}]\cong\mathbb{Z}_{N}[-q_{i}\text{ mod N}]$	$\mathbb{Z}_{6}\,[2,4,6]\mapsto\mathbb{Z}_{6}\,[4,2,6]$	$\neg$ (negated charges)

Table 2: The three certificate types , each worked through on a common

\mathbb{Z}_{6}

candidate; the full derivation follows below. Certificates 2 and 3 depend only on the

\mathbb{Z}_{N}

charges, independent of representations or VEVs. Every clause is

\neg(\text{the equivalent or excluded pattern})

; once added to the shared constraint database, it is enforced across all parallel searches with no physics evaluation spent on models it already rules out.

Space	Method	Models evaluated	Neutrino models found
$A_{4}\times\mathbb{Z}_{4}$	Random	4,000,000	28
	AMBer	4,000,000	683
	CDRL	1,000,000	1090
$A_{4}\times\mathbb{Z}_{N}$	Random	4,000,000	21
	AMBer	4,000,000	1394
	CDRL	1,000,000	2343

Table 4: Search Efficiency Comparison. CDRL discovers more neutrino models while evaluating significantly fewer candidates (up to

4\times

fewer) across all theory spaces, demonstrating substantially improved sample efficiency over AMBer and random search.

Figure 3: Distribution of successfully discovered neutrino models across $\mathbb{Z}_{N}$ symmetry groups for AMBer and CDRL. CDRL discovers viable models more uniformly across different symmetry choices, while AMBer concentrates more heavily around specific values of $N$ .

Figure 4: Distribution of discovered models over $\chi^{2}$ (fit quality) and $n_{p}$ (number of parameters). Each bin shows model density, with the bottom-left region corresponding to well-fitting, low-complexity models. CDRL places more mass in the low- $n_{p}$ areas. Within that region, $\chi^{2}$ differences at very low values do not indicate meaningfully different fit quality, so Table 3 and Table 5 remain the primary quantitative comparison, with this figure illustrating where in $(\chi^{2},n_{p})$ space that advantage is concentrated.

Panel of Figure 5 — Figure 5: Example decision trees. Left: Determination of the $E_{2}$ ’s $\mathbb{Z}_{4}$ charge in $A_{4}\times\mathbb{Z}_{4}$ . Among high-confidence states, this decision tree predicts the $E_{2}$ ’s $\mathbb{Z}_{4}$ charge from $E_{1}$ ’s $\mathbb{Z}_{4}$ charge. This demonstrates how CDRL has learned to match the $\mathbb{Z}_{4}$ charges of the charged lepton singlets, a reasonable result in order to construct physically realizable mass matrices. This observation also shows how some rules exclude possible choices (in this case the $\mathbb{Z}_{4}$ charge value of 4). Right: Determination of the $H_{u}$ ’s $A_{4}$ representation in $A_{4}\times\mathbb{Z}_{N}$ . This decision tree is harder to interpret physically. The up-flavor Higgs is not directly correlated with the charged lepton sector, so there is either some non-trivial relationship, or this is a spurious correlation within the CDRL search. The full set of such observations for all three theory spaces is in Appendix C .

\tilde{P}(a\mid s)\propto P(a\mid s)\,(1+\beta\,r_{a}(s))

Algorithm 2 Soft Constraint Extraction and Policy Reweighting

Figure 6: Impact of search knowledge discovery (KD) across theory spaces. Each panel is titled by metric and theory space. We compare CDRL with and without rule-based soft constraints on $A_{4}\times\mathbb{Z}_{4}$ , $A_{4}\times\mathbb{Z}_{N}$ , and $T_{19}\times\mathbb{Z}_{4}$ . Incorporating rules extracted from high-confidence search trajectories consistently improves both valid model rates and neutrino model discovery. Rule-guided runs achieve higher final performance and faster convergence, with gains of roughly $1.5$ – $2\times$ in valid models and $2$ – $3\times$ in neutrino models across the three spaces, demonstrating that the extracted rules provide useful and generalizable guidance for exploration. These experiments are performed using smaller-scale KD evaluation runs, so absolute values are not directly comparable to the full-scale results reported in Table 3 .

Panel of Figure 7 — Figure 7: The CDRL mechanism, shown as four snapshots of the same tree. (1) Start: the empty model; three of the nine allowed $\mathbb{Z}_{N}$ choices are shown at the root. (2) Propose, mask, descend: at the root, and again at every node MCTS visits, the BCP referee masks the illegal children first and the policy-value network then ranks what remains; MCTS commits to $\mathbb{Z}_{N}=6$ and descends, and the same red $\times$ masking recurs at each node along the way. Continued below.

Figure 8: (continued) (3) Two signals from one evaluation: the branch completes and the candidate fails, rank-deficient for every

\mathbb{Z}_{N}

charge; this single evaluation produces a scalar reward that retrains the network and, separately, a certificate that the analyzer converts into a clause added to the shared database. (4) Same pattern, different branch: MCTS next explores the sibling

\mathbb{Z}_{N}=4

branch, and the clause learned in step 3 masks the same representation-and-VEV pattern as soon as it recurs, before the branch is built out further or any physics evaluation is spent on it.

Explore similar work

OpenJev-RLCD: A Working RLCD Implementation
Sep 30, 2026Zhimin Gao, Pichao WangRL for Language Models RL for Language Model Reasoning
Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts
Sep 28, 2026Chenxiao Fan, Chongming Gao, Gangyi Zhang +8Confidence Estimation in Language Models RL for Language Model Reasoning
University of Science and Technology of China · Qwen Business Unit of Alibaba · National University of Singapore
Decoupling Exploration from Optimization in RLVR
Oct 7, 2026Saif Punjwani, Micah GoldblumRL for Language Model Reasoning RL Exploration
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University

Sep 30, 2026J/K move · Enter open · S save

OpenJev-RLCD: A Working RLCD Implementation

Zhimin Gao, Pichao Wang

RL for Language Models RL for Language Model Reasoning Language Model Calibration

Open paper AskPDF arXiv

Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive% of the items at

\le

5% error, versus \gvGrpoCovFive% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.

Abstract

Explore similar work

OpenJev-RLCD: A Working RLCD Implementation

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

Decoupling Exploration from Optimization in RLVR