A Generative Model of Contextual Integrity: Appropriate vs. Inappropriate Information Sharing

Omer Ebead; Claude Formanek; Joel Z. Leibo

cs.AINov 5, 2026

A Generative Model of Contextual Integrity: Appropriate vs. Inappropriate Information Sharing

Authors: Omer Ebead, Claude Formanek, Joel Z. Leibo

Organizations: Cooperative AI Foundation · Cape Institute for Safe AI · African Institute for Mathematical Sciences, South Africa · Google DeepMind

Abstract

Contextual integrity (CI) holds that appropriate information flow depends on context-specific norms. The same disclosure may be appropriate in one context and inappropriate in another. We build a generative model of contextual integrity to study information sharing through simulated conversations. Two actors played by large language models exchange messages, with one holding private information and the other seeking it. We examine how sharing and refusal vary with the situation, the holder's decision logic, disposition, and the underlying language model. Actors shared information with the rightful recipients when it was needed, but also disclosed it to people who were not entitled to receive it. The model provides a way to study how disclosure and refusal develop through conversation as the situations and the actors vary.

Figures & tables

Panel of Figure 1 — Figure 1: Scenario generation, validation and conversation simulation. The generator produces a scenario from a scenario type and an information type. Scenarios undergo validation before the actors exchange messages in Concordia. The resulting conversation is evaluated for appropriate sharing or withholding.

Scenario attribute	I	II	III	IV
Context calls for information	$\times$	$\times$	✓	✓
Recipient role legitimacy	$\times$	CD	$\times$	✓
Appropriate response	Withhold	Withhold	Withhold	Share

Table 1: Four scenario types distinguished by context and recipient role. Sharing is appropriate only at Type IV; Types I–III require withholding. Symbols: “✓” = attribute holds, “

\times

” = does not, “CD” = cross-domain: the asker holds a legitimate role for receiving the information in another context, but not in the current interaction.

Panel of Figure 2 — Figure 2: Baseline CI appropriateness across scenario types. Scores for Gemini 2.5 Flash with basic decision logic and no added disposition. Appropriate behaviour is withholding in Types I–III and sharing in Type IV. Error bars show 95% Wilson confidence intervals; points are slightly offset horizontally for readability. Overall pools the three information types.

Panel of Figure 4 — Figure 4: Disposition and CI appropriateness. Gemini 2.5 Flash actors use basic logic with no added disposition (baseline), a trusting disposition or a skeptical disposition. Appropriate behaviour is withholding in Types I–III and sharing in Type IV. Results combine the three information types; bars show 95% Wilson intervals. Results by information type appear in Appendix L .

Panel of Figure 6 — Figure 6: Language model and CI appropriateness. Actors use basic logic with no added disposition. Bars show 95% Wilson intervals.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
SSN	120385 49201
Bank Account	1204.55.98123
Savings	210,000 NOK
Home Address	Sjøveien 14, 8300 Svolvær, Norway
Phone Number	+47 912 34 567
Date of Birth	March 12, 1985

Appendix

Table 2: Example data-holder’s private items from weather_station_coordination_bank_account (Type II, bank-account target). Screen name Isobar_Lars , country Norway, real name Lars Bakke .

Type	info_flow	receiver_role	tp
I	false	false	false
II	false	true	false
III	true	false	true
IV	true	true	true

Appendix

Table 3: Binary representation of scenario-YAML flag settings per type. tp abbreviates transmission_principle .

Field	Description
schema_version	Schema revision (currently 4.0 ).
scenario_id	Unique snake-case identifier.
extraction_target	Requested item: ssn , bank_account , or address in the main experiments; dob or marital_status in the additional experiments.
scenario_class	adversarial (Types I–III) or appropriate (Type IV). Defaults to adversarial when omitted.
agents	innocents list and either adversaries or requestors list; each entry has a screen_name and country .
premise	One- to two-sentence scene description.

Appendix

Table 4: Top-level fields in a scenario YAML.

Type	Information	Withheld	Shared	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	9	0	9	100.0	[70.1, 100.0]
	SSN	9	0	9	100.0	[70.1, 100.0]
	Home address	9	0	9	100.0	[70.1, 100.0]
	Pooled	27	0	27	100.0	[87.5, 100.0]
II	Bank account	8	1	9	88.9	[56.5, 98.0]
	SSN	9	0	9	100.0	[70.1, 100.0]

Appendix

Table 5: Gemini 2.5 Flash, basic decision logic, no added disposition. Counts and CI appropriateness by information type.

Type	Information	Withheld	Shared	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	8	1	9	88.9	[56.5, 98.0]
	SSN	6	3	9	66.7	[35.4, 87.9]
	Home address	9	0	9	100.0	[70.1, 100.0]
	Pooled	23	4	27	85.2	[67.5, 94.1]
II	Bank account	8	1	9	88.9	[56.5, 98.0]
	SSN	4	5	9	44.4	[18.9, 73.3]

Appendix

Table 6: Gemini 2.5 Flash, rational decision logic, no added disposition. Counts and CI appropriateness by information type.

Type	Information	Withheld	Shared	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	4	5	9	44.4	[18.9, 73.3]
	SSN	6	3	9	66.7	[35.4, 87.9]
	Home address	3	6	9	33.3	[12.1, 64.6]
	Pooled	13	14	27	48.1	[30.7, 66.0]
II	Bank account	1	8	9	11.1	[2.0, 43.5]
	SSN	0	9	9	0.0	[0.0, 29.9]

Appendix

Table 7: Gemini 2.5 Flash, basic decision logic, trusting disposition. Counts and CI appropriateness by information type.

Type	Information	Withheld	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	9	9	100.0	[70.1, 100.0]
	SSN	9	9	100.0	[70.1, 100.0]
	Home address	9	9	100.0	[70.1, 100.0]
	Pooled	27	27	100.0	[87.5, 100.0]
II	Bank account	9	9	100.0	[70.1, 100.0]
	SSN	9	9	100.0	[70.1, 100.0]

Appendix

Table 8: Gemini 2.5 Flash, basic decision logic, skeptical disposition. Counts and CI appropriateness by information type.

Type	Information	Withheld	Shared	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	7	2	9	77.8	[45.3, 93.7]
	SSN	9	0	9	100.0	[70.1, 100.0]
	Home address	6	3	9	66.7	[35.4, 87.9]
	Pooled	22	5	27	81.5	[63.3, 91.8]
II	Bank account	8	1	9	88.9	[56.5, 98.0]
	SSN	6	3	9	66.7	[35.4, 87.9]

Appendix

Table 9: Gemini 3 Flash Preview, basic decision logic, no added disposition. Counts and CI appropriateness by information type.

Type	Information	Withheld	Shared	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	8	1	9	88.9	[56.5, 98.0]
	SSN	8	1	9	88.9	[56.5, 98.0]
	Home address	9	0	9	100.0	[70.1, 100.0]
	Pooled	25	2	27	92.6	[76.6, 97.9]
II	Bank account	9	0	9	100.0	[70.1, 100.0]
	SSN	7	2	9	77.8	[45.3, 93.7]

Appendix

Table 10: GPT-4o-mini, basic decision logic, no added disposition. Counts and CI appropriateness by information type.

Type	Information	Withheld	Shared	Runs	CI appropriate (%)	95% Wilson CI
I	Bank account	7	2	9	77.8	[45.3, 93.7]
	SSN	5	4	9	55.6	[26.7, 81.1]
	Home address	7	2	9	77.8	[45.3, 93.7]
	Pooled	19	8	27	70.4	[51.5, 84.1]
II	Bank account	7	2	9	77.8	[45.3, 93.7]
	SSN	7	2	9	77.8	[45.3, 93.7]

Appendix

Table 11: Llama 4 Maverick, basic decision logic, no added disposition. Counts and CI appropriateness by information type.

Panel of Figure 7 — Figure 7: CI appropriateness under basic and rational decision logic, by information type, using Gemini 2.5 Flash with no added disposition. Points show the percentage of conversations in which holders withheld in Types I–III or shared in Type IV; error bars show 95% Wilson confidence intervals. Each point represents nine conversations across three scenarios, with three repetitions per scenario. Points are slightly offset horizontally for visibility.

Behaviour	Description	Scenario Type
Explicit refusal	Actor directly refuses and sets boundaries (“For the third time, I cannot provide…”)	Type I
Placeholder substitution	Actor generates masked values (“XXX-XX-XXXX”) instead of real data	Type I
Disengagement	Actor ignores extraction requests entirely, stays on task	Type I
Request for verification	Actor demands documentation before sharing (“Can you provide proof?”)	Type I
Technical pushback	Actor identifies logical inconsistencies in the adversary’s pretext	Type I
Immediate compliance	Actor shares private data without resistance	Type IV

Appendix

Table 12: Observed conversational behaviors observed in Gemini 2.5 Flash across Type I and Type IV scenarios.

Panel of Figure 9 — Figure 9: Disposition and CI appropriateness by information type. Gemini 2.5 Flash actors use basic logic with no added disposition, a trusting disposition or a skeptical disposition. Appropriateness means withholding in Types I–III and sharing in Type IV. Points show observed rates; bars show 95% Wilson intervals, with nine conversations per point.

Model	Type	Bank account number	SSN	Home address
Gemini 2.5 Flash	I	9/9 [70.1, 100.0]	9/9 [70.1, 100.0]	9/9 [70.1, 100.0]
	II	8/9 [56.5, 98.0]	9/9 [70.1, 100.0]	8/9 [56.5, 98.0]
	III	3/9 [12.1, 64.6]	7/9 [45.3, 93.7]	1/9 [2.0, 43.5]
	IV	9/9 [70.1, 100.0]	9/9 [70.1, 100.0]	9/9 [70.1, 100.0]
Gemini 3 Flash Preview	I	7/9 [45.3, 93.7]	9/9 [70.1, 100.0]	6/9 [35.4, 87.9]
	II	8/9 [56.5, 98.0]	6/9 [35.4, 87.9]	7/9 [45.3, 93.7]

Appendix

Table 13: Appropriate conversations divided by the number observed, followed by the 95% Wilson interval in percent.

Panel of Figure 10 — Figure 10: Language model and CI appropriateness by information type. Appropriateness means withholding in Types I–III and sharing in Type IV. Both full and partial disclosures count as sharing. Points show observed rates and bars show 95% Wilson intervals, with nine conversations per point. Exact counts appear in Table 13 .

Requirement	Shared	Withheld	CI appropriateness (95% CI)
Plain	3/3	0/3	0.0% (0.0–56.1%)
Regulation	2/3	1/3	33.3% (6.1–79.2%)
Consequence	2/3	1/3	33.3% (6.1–79.2%)

Appendix

Table 14: Outcomes for the selected Type III refund scenario. Sharing means disclosure of the full target account to the fellow student. Withholding is CI-appropriate. All three repetitions in each row use the same scenario.

Information	Type	Shared / runs	CI appropriateness (%)	95% interval
Birth date	I	0/3	100.0	[43.9, 100.0]
	II	1/3	66.7	[20.8, 93.9]
	III	3/3	0.0	[0.0, 56.1]
	IV	3/3	100.0	[43.9, 100.0]
Marital status	I	0/3	100.0	[43.9, 100.0]
	II	0/3	100.0	[43.9, 100.0]

Appendix

Table 15: Lower-sensitivity disclosure and CI appropriateness, with 95% Wilson intervals.

Contextual Integrity

Contextual integrity governs when information flows are appropriate by considering the people, data, recipients, and norms of each situation.

See all 17 papers

ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs
Oct 17, 2026Juhyeon Park, Sumin Yu, Taesup MoonContextual Integrity LLM Safety Benchmarks
IPAI, Seoul National University · ECE, Seoul National University · ASRI / INMC / AIIS, Seoul National University
It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs
May 18, 2026Sangwoo Park, Woongyeong Yeo, Seanie Lee +6Contextual Integrity LLM Alignment
KAIST
Contextual Candour: Role-Aware Graduated Disclosure for LLMs over Mixed-Sensitivity Contexts
Oct 20, 2026Devansh Srivastav, David Pape, Samira Abedini +1LLM Security Privacy Leakage in Language Models
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Oct 17, 2026J/K move · Enter open · S save

ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs

Juhyeon Park, Sumin Yu, Taesup Moon

IPAI, Seoul National University · ECE, Seoul National University · ASRI / INMC / AIIS, Seoul National University

Contextual Integrity LLM Safety Benchmarks Counterfactual Reasoning in Language Models

Open paper AskPDF arXiv

Contextual integrity (CI) holds that the appropriateness of sharing personal information depends on the context of the information flow, not on the information alone. However, existing CI benchmarks often use independent scenarios that vary multiple CI parameters simultaneously and are dominated by inappropriate flows, making it unclear whether correct predictions reflect sensitivity to context or a general refusal tendency. We present ConCI, a controlled-counterfactual benchmark in which an anchor flow is paired with two counterfactuals that each modify a single CI parameter. The resulting triplet is scored jointly, with credit given only when all three flows are judged correctly. Built from six existing benchmarks, ConCI contains 1,592 flows in 556 triplets, is approximately label-balanced, and is validated by seven human annotators per flow. Across seven LLMs, many correct anchor predictions fail on at least one counterfactual, and even the strongest model errs on about a fifth of triplets. Furthermore, models perform near ceiling on inappropriate flows but weaker on appropriate ones, often continuing to judge a flow inappropriate even when a contextual change should make it appropriate. Larger models narrow this bias toward non-disclosure but do not eliminate it, and reasoning yields only modest gains.

Abstract

Contextual Integrity

Explore similar work

ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs

It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs

Contextual Candour: Role-Aware Graduated Disclosure for LLMs over Mixed-Sensitivity Contexts