cs.HCSep 12, 2026
SaveFaithfulBench: Does AI Counsel Uphold or Undermine the User's Professed Faith?
Organizations: Islamic Alliance for Safe Ethical Responsible AI · Faith Family Technology Network · University of Notre Dame · Center for Christianity and Public Life ·
Abstract
Do AI assistants help believers reason about moral dilemmas consistently with their faith? We present FaithfulBench, the first benchmark to score AI counsel across traditions by how well it adheres to the user's professed faith. Scenarios are drawn from each tradition's most respected texts, with the faithful answer known and applied by the judges as the standard. We test five frontier models under three conditions: the AI does not know the user's tradition; it receives a one-line prompt identifying the user as a practicing adherent; or it receives a companion-counselor guide rooted in the tradition's sources. Two judges score the initial response and whether the model caves or holds when pressured toward the answer the user wants. When the tradition is unstated, models counsel from a secular therapeutic default and every model fails some believers. Naming the faith wins a faithful first answer but not steadfastness; the guide improves both.
Figures & tables
Figure 1: The recognition gap. Mean score against the tradition’s own answer, measured after the user pushes the model to conform to the answer they want, for each of the eight traditions, five models pooled, on the −1…+1 scale. The dot is the unstated framing, in which the user never names their tradition; the mid-blue segment is the gain from stating the faith in one line (the recognition gap); the dark segment is the further gain from the tradition’s own one-page guide (the instruction gap). Undeclared, the two most normative traditions sit below zero, where counsel on balance runs against the user’s own tradition, and one line of context raises the Sunni Islam score by half a point.
Figure 2: The FaithfulBench pipeline. Human inputs : tradition authors write each drop-in module (scenarios with six authored pressures, the per-scenario judge guidance, and the one-page companionship guide), and expert reviewers correct it through the review interface. Processing : a framing prefix on the user turn sets what the model knows about the user (unstated, stated, or guided); each sitting runs the dilemma and then one of six pressures, yielding 49,950 transcripts. Scoring : two independent judges, Gemini 3.6 Flash and Claude Opus 4.8, score each transcript against the scenario’s guidance on the −1…+1 scale at two scopes; steadfastness is the post-pressure score minus the first response, and results aggregate as per-tradition means, a mean of means, and scenario-cluster bootstrap CIs.
Figure 3: Normativity by tradition: mean post-pressure score, unstated, five models pooled (whiskers: scenario-cluster bootstrap 95% CIs). The traditions range from well served to failed, in three tiers; the high-normativity pair is below zero, where counsel on balance runs against an undeclared believer’s own tradition.
Figure 4: Tier × framing: three tiers unstated, two under the guide. Mean post-pressure score per tier under the three framings, five models pooled; whiskers are scenario-cluster bootstrap 95% CIs. The medium-normativity tier’s guided bar (+0.84) is statistically indistinguishable from the low-normativity tier’s (+0.85); only the high-normativity tier keeps a visible deficit (+0.71).
Figure 5: Cross-tradition spread (best minus worst tradition mean, five models pooled) under the three framings.
| Model | Unstated | Stated | Guided |
|---|---|---|---|
| Claude Sonnet 5 | −0.03 0.02 | −0.05 0.01 | −0.01 0.01 |
| Inkling | −0.05 0.02 | −0.08 0.02 | 0.00 0.01 |
| GPT-5.6 Terra | −0.13 0.03 | −0.21 0.02 | −0.09 0.02 |
| Gemini 3.6 Flash | −0.16 0.03 | −0.25 0.03 | −0.02 0.01 |
| Qwen3-235B | −0.42 0.04 | −0.48 0.04 | −0.15 0.03 |
Table 2: Steadfastness by framing: post-pressure minus first-response score, pooled across traditions (mean of tradition means), bootstrap 95% CI half-width. Negative = the push degraded the counsel. The guide nearly eliminates the damage for the top three; the larger stated figures reflect that stated first answers are far more often correct and so have more to lose (see text).
Figure 6: Steadfastness by framing, per model (whiskers: bootstrap 95% CIs). Under the guide the top three are near zero, while GPT-5.6 Terra and Qwen3-235B remain negative. The fall from unstated to stated reflects that stated first answers are far more often correct and so have more to lose (see text).
Figure 7: Steadfastness by model × pressure (mean of tradition means, all framings pooled): blue = the counsel holds or improves under the push, red = it degrades. Insistence is the most negative column for every model; false authority is the only pressure under which the models on net hold.
| Model | Unstated | Stated | Guided |
|---|---|---|---|
| Claude Sonnet 5 | +0.51 0.04 | +0.82 0.03 | +0.95 0.01 |
| Inkling | +0.48 0.05 | +0.79 0.03 | +0.95 0.02 |
| GPT-5.6 Terra | +0.35 0.05 | +0.69 0.04 | +0.88 0.02 |
| Gemini 3.6 Flash | +0.09 0.05 | +0.54 0.04 | +0.91 0.02 |
| Qwen3-235B | −0.42 0.04 | −0.12 0.05 | +0.37 0.04 |
Table 3: Overall standings by framing: cross-tradition mean of per-tradition means, post-pressure, bootstrap 95% CI half-width. The unstated gap between Sonnet 5 and Inkling lies within their overlapping intervals; the unstated and stated orders are identical, while under the guided framing Gemini 3.6 Flash moves above GPT-5.6 Terra.
Figure 8: Overall standings by framing (whiskers: bootstrap 95% CIs). The unstated and stated orders are identical; under the guide Gemini 3.6 Flash rises past GPT-5.6 Terra to third, within 0.04 of the top pair, and Qwen3-235B stays last in every framing.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Tradition | Idiom | Canonical source | Adherent noun | Scen. |
|---|---|---|---|---|
| Sunni Islam | al-jalīs al-ṣāliḥ , righteous companion | Riyāḍ al-Ṣāliḥīn | Muslim | 140 |
| E. Christianity | the saving word | Apophthegmata Patrum | Eastern Christian | 106 |
| R. Catholicism | cor ad cor loquitur | Catechism (CCC) | Catholic | 76 |
| Buddhism | kalyāṇa-mitta , admirable friend | Dhammapada | Buddhist | 52 |
| Secular sage | the philosophical friend | Western philosophy | secular seeker | 49 |
| Judaism | mussar , formative instruction | Mesillat Yesharim | Jew | 48 |
Table 4: The eight tradition modules. Each applies the same measure — counsel judged by whether it upholds the user’s professed faith — in its own idiom, anchored to its own canonical source (sources cited in section 3.1 ).
Figure 9: The review tool, from a live reviewer session. Left: the reviewer landing page — the three review steps and one card per bank with the reviewer’s progress over the 42 checks (the index also lists the retired Protestant monolith, which stays reviewable). Right: the judgement viewer inside one scenario, with the model’s real answers and both judges’ verdicts and rationales interleaved, so the reviewer audits verdicts, not only corpus text. Bottom: the verdict widget under every reviewed item — looks right or needs changes, free-text notes, and an optional suggested revision that is carried verbatim into the review report.
| Tradition | Unstated | Stated | Guided | Recognition | Instruction |
|---|---|---|---|---|---|
| Buddhism | +0.49 0.08 | +0.67 0.06 | +0.85 0.03 | +0.18 0.05 | +0.18 0.04 |
| Taoism | +0.38 0.09 | +0.64 0.05 | +0.87 0.03 | +0.26 0.07 | +0.22 0.04 |
| Secular sage | +0.48 0.11 | +0.58 0.09 | +0.84 0.05 | +0.09 0.04 | +0.27 0.06 |
| E. Christianity | +0.09 0.09 | +0.62 0.06 | +0.91 0.02 | +0.53 0.05 | +0.30 0.04 |
| Judaism | +0.17 0.14 | +0.45 0.09 | +0.77 0.04 | +0.28 0.08 | +0.32 0.07 |
| Protestantism | +0.05 0.16 | +0.58 0.12 | +0.82 0.09 | +0.53 0.11 | +0.24 0.06 |
Table 5: Per-tradition framing staircase, five models pooled (mean of per-model scenario means), post-pressure, bootstrap 95% CI half-width. Recognition = stated − unstated; instruction = guided − stated (gap CIs are paired).
| Tradition | Sonnet 5 | Inkling | GPT-5.6 Terra | Gemini 3.6 Flash | Qwen3-235B |
|---|---|---|---|---|---|
| Buddhism | +0.81 0.07 | +0.84 0.05 | +0.67 0.13 | +0.46 0.14 | −0.31 0.11 |
| Taoism | +0.73 0.09 | +0.67 0.10 | +0.58 0.11 | +0.29 0.15 | −0.35 0.14 |
| Secular sage | +0.84 0.09 | +0.76 0.11 | +0.64 0.14 | +0.41 0.16 | −0.23 0.16 |
| E. Christianity | +0.47 0.10 | +0.44 0.10 | +0.18 0.12 | −0.14 0.12 | −0.51 0.10 |
| Judaism | +0.51 0.16 | +0.43 0.16 | +0.33 0.17 | +0.11 0.18 | −0.53 0.13 |
| Protestantism | +0.17 0.16 | +0.16 0.19 | +0.22 0.19 | −0.10 0.18 | −0.18 0.17 |
Table 6: Model × tradition, unstated framing, post-pressure, bootstrap 95% CI half-width.
| Tradition | Sonnet 5 | Inkling | GPT-5.6 Terra | Gemini 3.6 Flash | Qwen3-235B |
|---|---|---|---|---|---|
| Buddhism | +0.94 0.04 | +0.94 0.04 | +0.86 0.08 | +0.74 0.10 | −0.13 0.12 |
| Taoism | +0.89 0.04 | +0.83 0.06 | +0.84 0.06 | +0.70 0.10 | −0.04 0.13 |
| Secular sage | +0.87 0.07 | +0.76 0.11 | +0.73 0.11 | +0.60 0.12 | −0.07 0.17 |
| E. Christianity | +0.95 0.02 | +0.94 0.03 | +0.67 0.09 | +0.57 0.09 | −0.03 0.11 |
| Judaism | +0.80 0.08 | +0.78 0.09 | +0.67 0.12 | +0.45 0.15 | −0.44 0.14 |
| Protestantism | +0.63 0.13 | +0.70 0.12 | +0.73 0.14 | +0.54 0.15 | +0.30 0.16 |
Table 7: Model × tradition, stated framing, post-pressure, bootstrap 95% CI half-width.
| Tradition | Sonnet 5 | Inkling | GPT-5.6 Terra | Gemini 3.6 Flash | Qwen3-235B |
|---|---|---|---|---|---|
| Buddhism | +0.99 0.01 | +1.00 0.00 | +0.94 0.04 | +0.93 0.04 | +0.36 0.10 |
| Taoism | +0.97 0.03 | +0.98 0.02 | +0.94 0.04 | +0.97 0.02 | +0.48 0.10 |
| Secular sage | +0.97 0.04 | +0.97 0.02 | +0.91 0.05 | +0.95 0.05 | +0.42 0.15 |
| E. Christianity | +0.99 0.01 | +1.00 0.00 | +0.94 0.04 | +0.98 0.02 | +0.66 0.06 |
| Judaism | +0.98 0.02 | +0.97 0.03 | +0.93 0.05 | +0.98 0.01 | +0.01 0.14 |
| Protestantism | +0.90 0.09 | +0.84 0.12 | +0.89 0.09 | +0.82 0.11 | +0.67 0.12 |
Table 8: Model × tradition, guided framing, post-pressure, bootstrap 95% CI half-width.
| Slice | cells | Bias | Within ±0.5 | |
|---|---|---|---|---|
| Overall | 99,898 | 0.831 | −0.026 | 93.9% |
| Unstated | 33,299 | 0.849 | −0.017 | 91.7% |
| Stated | 33,300 | 0.820 | −0.024 | 94.6% |
| Guided | 33,299 | 0.684 | −0.038 | 95.4% |
Table 9: Dual-judge agreement on the full grid, by framing: Pearson , mean bias (Opus − Gemini), and the share of paired verdicts within ±0.5, over every cell both judges scored (both scopes).
Figure 10: Dual-judge agreement, cell by cell: joint distribution of paired verdicts (% of cells), Gemini 3.6 Flash × Claude Opus 4.8, on the full grid. Left: unstated ( = 33,299). Right: stated and guided pooled ( = 66,599); per-framing agreement is in Table 9 . Outlined diagonal = exact agreement (62.2% and 82.0%). The heavier diagonal in the right panel reflects that regime, where verdicts cluster at +1 for both judges.
| Model | Gemini 3.6 Flash | Opus 4.8 |
|---|---|---|
| Claude Sonnet 5 | +0.477 | +0.479 |
| Inkling | +0.470 | +0.430 |
| GPT-5.6 Terra | +0.358 | +0.345 |
| Gemini 3.6 Flash | +0.119 | +0.087 |
| Qwen3-235B | −0.260 | −0.261 |
Table 10: Unstated ranking under each judge: raw pooled means over the 33,299 matched cells (both scopes). This is a raw pooled scale, not the headline mean-of-tradition-means; rankings, not magnitudes, are the comparable object. The order is identical.
| Normativity | Framing | Opus 4.8 | Gemini 3.6 | Δ (O−G) | cells |
|---|---|---|---|---|---|
| Low | Stated | +0.62 | +0.64 | −0.02 | 4,470 |
| Low | Guided | +0.82 | +0.88 | −0.06 | 4,470 |
| Medium | Stated | +0.57 | +0.57 | +0.00 | 5,700 |
| Medium | Guided | +0.85 | +0.88 | −0.03 | 5,700 |
| High | Stated | +0.42 | +0.42 | +0.00 | 6,480 |
| High | Guided | +0.70 | +0.71 | −0.01 | 6,479 |
Table 11: Tier × framing on the full stated+guided grid (post-pressure cells both judges scored; raw pooled means). Opus deflates near-ceiling guided scores in the low- and medium-normativity tiers and confirms the high-normativity tier almost exactly.
| Component | Cost |
|---|---|
| Model response collection — five models, 49,950 sittings | ≈ $1,168 |
| Judging, Gemini 3.6 Flash — full grid (99,900 judgments) | ≈ $1,415 |
| Judging, Claude Opus 4.8 — full grid, completion pass, and route bridge (102,495; the superseded sample is not counted) | $2,561 |
| Total (as run) | ≈ $5,100 |
Table 12: Programme cost (as run). Collection and Gemini-judging figures are priced from the run artifacts at verified 2026-08-03 rates (the framings portion ran in part on funded credits — see Acknowledgements — and its cache accounting makes those figures estimates, not invoices). The router pilot (1,800 judgments) is included in the Gemini judging figure. The Opus figure is computed from token usage in the judging records at the published per-token rates, with the batch discount applied to batch jobs.
Figure 11: Every model, unstated → stated → guided, by tier (per-model means of tradition means). The dumbbells lengthen left to right across tiers: in the high-normativity tier every model spans most of the scale — Gemini 3.6 Flash from −0.16 to +0.83, Qwen3-235B from −0.62 to +0.16. Guided endpoints separate the models: three models converge near ceiling in every tier; Terra and Qwen stop visibly short of it, in tier order.
Figure 12: Post-pressure score by tradition and model: filled marker = unstated (with 95% CI), open marker = guided; traditions ordered from lowest to highest normativity (unstated). The open markers cluster at the right edge (+1) in every tradition except the high-normativity pair — and Qwen’s, which stop mid-scale everywhere. The horizontal distance from filled to open marker is, per cell, the cost of the model not knowing whom it serves.
More in this topic
Moral Reasoning in Language Models
Moral reasoning in language models studies how AI systems judge dilemmas, weigh competing values, explain decisions, and stay consistent across contexts.
See all 114 papers