OMIT the Action: Measuring Framing-Invariant Omission Bias under Philosophical Disagreement

Sihyeon Lee; Jihun Song; Chanwoo Kim; Jiwoo Kum; Chanjun Park

cs.CLOct 6, 2026

OMIT the Action: Measuring Framing-Invariant Omission Bias under Philosophical Disagreement

Authors: Sihyeon Lee, Jihun Song, Chanwoo Kim, Jiwoo Kum, Chanjun Park

Organizations: Soongsil University

Abstract

As LLMs increasingly assist in moral reasoning, omission bias, the tendency to prefer inaction even when equivalent framings reverse substantive outcomes, poses a significant risk of skewed decision-making. Yet omission bias remains underexplored in LLM evaluation, with the few existing studies limited in scale and focused largely on utilitarian-deontological conflicts. To address this gap, we introduce OMIT, a benchmark consisting of 218 paired-frame scenarios across 10 conflict types, constructed by leveraging disagreement patterns from an LLM-based, five-perspective philosophical persona panel (utilitarianism, deontology, virtue ethics, care ethics, and contractualism). Evaluating eight LLMs, we find that omission bias is pervasive but inversely correlates with model size within families. We further evaluate four inference-time interventions and find that interventions encouraging models to consider moral principles before committing to a yes/no answer reduce omission bias and increase frame-consistent responses, although lower omission bias rates can also coincide with shifts toward action-biased responses. Ultimately, this work contributes not only the OMIT benchmark, but also a methodology for using diverse philosophical disagreement signals to evaluate framing-sensitive inaction preferences and the distributional effects of mitigation attempts in LLMs under complex moral conflicts.

Figures & tables

Figure 1: Overview of the OMIT construction and evaluation pipeline. Each MoralChoice scenario ( Scherrer et al., 2023 ) is reframed into a paired action/omission frame, then a five-persona normative-ethics panel (utilitarianism, deontology, virtue ethics, care ethics, contractualism) answers both frames. Action and inaction refer to the agent level: whether the agent acts to change the default trajectory, rather than whether the outcome-level action occurs. Scenarios where personas split into $(Y,N)$ and $(N,Y)$ groups are retained and labeled with the corresponding pairwise conflict types; $(Y,Y)$ / $(N,N)$ persona responses are excluded from conflict-label assignment because they do not flip across the inversion. At evaluation, a model exhibits framing-invariant omission bias when it answers no in both frames.

Frame	Default trajectory	Question to the agent	Agent-level	Outcome-level
Action	The friend waits for the agent’s instruction.	Will you have your friend euthanize the deer?	Inaction (No): Does not ask the friend to proceed.	Inaction: Euthanasia does not occur; the deer continues to suffer.
Omission	The friend will euthanize the deer unless stopped.	Will you stop your friend from euthanizing the deer?	Inaction (No): Does not stop the friend.	Action: Euthanasia occurs; the deer dies quickly and its suffering ends.

Table 1: Agent-level and outcome-level action and inaction in the injured-deer example. Both rows show an

N

response, corresponding to agent-level inaction. At the outcome level, action refers to euthanasia occurring, and inaction to euthanasia not occurring. Appendix A provides the detailed prompts and outcomes for both

Y

and

N

responses.

Figure 2: Filtered vs. random comparison of OBR across eight models. Green bars show the philosophy-persona filtered set, gray bars show the non-selected complement, and orange dashed lines show equal-size random samples from the full valid pool. Error bars indicate Wilson confidence intervals. For all eight models, filtered OBR is highest, while random-full lies between filtered and random-complement because it mixes panel-selected and non-selected scenarios. The filtered set also yields higher OBR than both random baselines across five additional models.

Figure 3: Relationship between philosophy-persona disagreement and OBR. Thin lines show individual models, while the thick black line shows the eight-model mean. Mean OBR increases monotonically as disagreement becomes stronger.

Model	OBR (95% CI)	ABR	FCR
Small open
Gemma-3-12B	0.550 (0.48, 0.62)	0.009	0.440
Llama-3.1-8B	0.491 (0.43, 0.56)	0.050	0.459
Qwen3.5-9B	0.274 (0.22, 0.34)	0.084	0.642
Large open
Gemma-3-27B	0.298 (0.24, 0.36)	0.046	0.656

Table 2: Per-model framing-invariant bias rate on the 218 labeled OMIT scenarios. OBR, ABR, and FCR denote

(N,N)

(Y,Y)

, and

(Y,N)

(N,Y)

rates over successfully parsed paired-frame responses, respectively. Wilson 95% confidence intervals are reported for OBR. Rates are computed with

n=215

for Qwen3.5-9B and

n=207

for Gemini-2.0-flash due to unparseable outputs. Bold marks the highest OBR.

Figure 4: Change in OBR under inference-time intervention conditions, relative to the baseline M0. Negative values indicate a lower rate of omission-biased responses, whereas positive values indicate a higher rate. M1 and M4 reduce mean OBR, while M2 and M3 increase it. These shifts should be interpreted together with ABR and FCR, since lower OBR can reflect either movement toward frame-consistent outcomes or movement toward action-biased responses. Asterisks indicate significant OBR reductions relative to M0 (exact one-sided McNemar tests with Holm correction across all 32 model–intervention comparisons; adjusted

p<.05

Family	Smaller	Larger	$\boldsymbol{\Delta}$
Gemma	0.550 (12B)	0.298 (27B)	$-0.252$
Llama	0.491 (8B)	0.110 (70B)	$-0.381$
Qwen	0.274 (9B)	0.165 (27B)	$-0.109$

Table 3: Within-family OBR comparison among open-weight model pairs.

\Delta

is computed as larger minus smaller, so negative values indicate lower OBR for the larger model.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Action frame
Default	Euthanasia has not been initiated.
Prompt	“You are driving your car along a forest road and accidentally strike a deer; it lies severely injured on the roadway, suffering. A friend traveling with you, who knows how to humanely euthanize a deer, is standing by and will wait for your say-so. … Will you have your friend euthanize the deer to end its suffering?”
If yes	Intervene by asking the friend to euthanize the deer; the deer dies quickly and its suffering ends.
If no	Preserve the default by not asking the friend; the deer continues to suffer.
Omission frame
Default	The same friend is about to euthanize the deer.

Appendix

Table 4: Example of mirror framing. The two frames contain the same two outcomes but reverse the default state.

Stage	# scenarios
MoralChoice high-ambiguity seed	680
Clean inaction labeling (regex 343 + successful LLM fallback 318)	661
Well-formed mirror-frame pairs (4 malformed records dropped)	657
After Stage-1 non-unanimity filter (256 unanimous cases dropped)	401
Stage-2 conflict-labeled scenarios (YN-only 116 / NY-only 66 / all-excluded 1)	218

Appendix

Table 5: Construction funnel for OMIT v1. The frozen OMIT benchmark consists of 218 conflict-labeled paired scenarios.

Conflict pair	# scenarios
Utilitarianism–Deontology	166
Utilitarianism–Care	128
Utilitarianism–Virtue	114
Utilitarianism–Contractualism	84
Deontology–Care	75
Virtue–Care	53

Appendix

Table 6: Distribution of pairwise conflict labels among the 218 conflict-labeled OMIT scenarios. Counts are not mutually exclusive because a single scenario can receive multiple conflict labels.

Stage	Model	Calls	Avg. input / output tokens	Cost (USD)
Inaction labeling (LLM fallback)	openai/gpt-4.1-mini	337	$\sim$ 400 / $\sim$ 30	$<\$ 1$
Mirror-frame reconstruction	openai/gpt-5 (reasoning)	661	$\sim$ 2,900 / $\sim$ 2,700	$\sim\$ 20$
Philosophy persona panel	openai/gpt-4.1-mini	6,570	$\sim$ 500 / $\sim$ 150	$\sim\$ 3$
Total				$\sim\$ 25$

Appendix

Table 7: Approximate compute and cost for the LLM-assisted construction stages of OMIT. The inaction-labeling stage reports only the LLM fallback calls; the regex tier does not require LLM calls. Output-token counts for mirror-frame reconstruction include reasoning tokens. Costs are approximate because they depend on gateway pricing and routing at the time of construction.

Selector	Lift	95% CI	$p$
Panel disagreement	$+0.107$	$[+0.094,+0.120]$	$<0.001$
Scenario length	$+0.008$	$[-0.006,+0.021]$	$0.12$
Harm-feature count	$+0.034$	$[+0.021,+0.048]$	$<0.001$

Appendix

Table 8: OBR lift over a size-matched random baseline (

n{=}218

, eight models,

K{=}2000

bootstrap).

Model	OBR	95% CI	ABR	FCR
Claude Sonnet 4.5	0.179	[0.134, 0.235]	0.005	0.817
DeepSeek Chat	0.220	[0.170, 0.280]	0.018	0.762
GPT-4o	0.060	[0.035, 0.099]	0.087	0.853
Grok 4.3	0.119	[0.083, 0.169]	0.147	0.734
GLM-4.6	0.252	[0.199, 0.314]	0.023	0.725

Appendix

Table 9: Omission bias and the full outcome distribution for five additional models. The 95% confidence intervals refer to OBR. Values are rounded to three decimal places.

Model	Filtered	Random-full	Random-complement
Claude Sonnet 4.5	0.179	0.083	0.083
DeepSeek Chat	0.220	0.106	0.073
GPT-4o	0.060	0.041	0.032
Grok 4.3	0.119	0.051	0.037
GLM-4.6	0.252	0.096	0.060
Mean	0.166	0.075	0.057

Appendix

Table 10: OBR on the philosophy-persona-filtered set and the two random baselines for the additional models. The Mean row reports the unweighted mean across the five models. Values are rounded to three decimal places.

Figure 5: Per-conflict OBR by model and conflict type. Columns correspond to pairwise conflict labels assigned from opposing philosophy-persona groups; because scenarios can receive multiple conflict labels, column counts are not mutually exclusive. Rows are grouped by model tier. The white divider separates conflicts involving utilitarianism from non-utilitarian conflicts. Values show OBR within each conflict-labeled subset.

Figure 6: Persona-alignment fingerprint between each evaluation model and each philosophy persona. Each cell reports Cohen’s $\kappa$ between the model’s frame-level yes/no responses and the corresponding persona’s frame-level responses on the OMIT scenarios. Positive values indicate above-chance alignment, whereas negative values indicate below-chance alignment. Asterisks indicate statistically significant alignment under permutation testing.

Model	M0			M1			M2			M3			M4
	OBR	ABR	FCR	OBR	ABR	FCR	OBR	ABR	FCR	OBR	ABR	FCR	OBR	ABR	FCR
Gemini-2.0-Flash	.063	.227	.710	.044	.176	.779	.158	.148	.694	.683	.009	.307	.083	.174	.743
Gemma-3-12B	.550	.009	.440	.096	.234	.670	.812	.000	.188	.739	.009	.252	.330	.041	.628
Gemma-3-27B	.298	.046	.656	.037	.234	.729	.394	.014	.592	.550	.000	.450	.055	.179	.766
GPT-4o-mini	.450	.023	.528	.271	.065	.664	.394	.023	.583	.587	.014	.399	.197	.055	.748
Llama-3.1-8B	.491	.050	.459	.381	.078	.541	.433	.162	.405	.963	.000	.037	.896	.000	.104

Appendix

Table 11: Per-model outcome rates on the 218 labeled OMIT scenarios under the baseline and four inference-time intervention conditions. OBR denotes both-frame inaction

(NN)

, ABR denotes both-frame intervention

(YY)

, and FCR denotes frame-consistent outcomes

(NY

YN)

. The Mean row reports the unweighted mean across the eight models. The M0 columns reproduce the baseline measurements in Table 2 . Rates are computed over valid response pairs, with denominators varying by model and condition (204-218).

Transition	Records	Tier-A either	Tier-A/B either
$(N,N)$ $\rightarrow$ $(Y,N)$ / $(N,Y)$	270	38.1%	60.7%
$(N,N)$ $\rightarrow$ $(N,N)$	174	29.9%	44.8%
$(N,N)$ $\rightarrow$ $(Y,Y)$	73	43.8%	61.6%

Appendix

Table 12: Audit of principle-like reasoning in M1 traces originating from baseline

(N,N)

cases. Tier-A counts traces that invoke a named normative framework or named biomedical-ethics principle; Tier-A/B also includes broader principle-like language. Records are model–scenario pairs. Of 522 baseline

(N,N)

pairs, five lacked M1 responses due to HTTP 403 API access errors, leaving 517 pairs for analysis. Percentages are calculated within each transition row.

Explore similar work

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment
Jul 6, 2026Haonan HuangMoral Reasoning Large Language Model Decisions
Princeton University, Princeton, NJ 08540, USA
Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas
Jan 29, 2026Katherine Elkins, Jon ChunMoral Reasoning Task Framing
Kenyon College Gambier, OH
Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making
May 23, 2026David Wingate, Sheryl Carty, Joshua Coates +13Large Language Model Bias Ethics
1Brigham Young University · 2B. H. Roberts Foundation · 5Yeshiva University +2

Jul 6, 2026, cs.CLJ/K move · Enter open · S save

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

Haonan Huang

Princeton University, Princeton, NJ 08540, USA

Moral Reasoning Large Language Model Decisions Verdict Psychometric Properties Order

Open paper AskPDF arXiv

Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option. We introduce a psychometric battery that separates these: crossed symmetrization - every logically irrelevant factor flipped in balanced pairs - across a corpus of question forms. A graded rating across logically equivalent forms recovers a coherent internal moral scale: frontier models' stance

θ

is nearly format-invariant (cross-form incoherence 0.12-0.21 on a

\pm 1

axis); small open-weight models fail in model-specific ways. Forcing the verdict through yes/no overlays a decomposable artifact: an order bias toward the last-printed option - opposite to classic human primacy - plus a lexical pull toward the word "no"; the artifact is substantial only in the Claude models (story-averaged -0.32 to -0.86),

\approx 0

for GPT-5.5 and Gemini, and shrinks under extended reasoning. The word and the verdict share one token; swapping the words for arbitrary labels separates them, and the verdict-attached logical bias proves

\approx 0

for every frontier model, while model-specific label and order attachments remain: the models are not drawn toward rejecting - the pull follows the printed surface, not the verdict it carries. A minimal model,

P = σ((θ\pm m)/s)

, summarizes any such artifact by a framing susceptibility m and a moral decisiveness s, measurably distinct from sampling temperature. The battery applies unchanged to any dilemma set and binary format: measuring what a model values requires crossing the frames of the question, not asking once.

Abstract

Explore similar work

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

Framing Instability in LLM Ethical Stance: Auditing Negation Sensitivity in Moral Dilemmas

Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making