From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

Minming Ye; Min Liu; Haoang Chi; Ciheng Wu; Naiyang Guan

cs.CLOct 15, 2026

From Observation to Intervention: Distilling Self-Generated In-Context Demonstrations via Executable Causal Intervention

Authors: Minming Ye, Min Liu, Haoang Chi, Ciheng Wu, Naiyang Guan

Organizations: Shanghai Jiaotong University · Academy of Military Science · National University of Defense Technology

Abstract

Self-generated In-Context Learning (ICL) enables LLMs to synthesize their own few-shot demonstrations, often outperforming manual prompts. However, existing methods filter demonstrations by observation, assuming a rationale is high-quality if it leads to the correct output. We show such filtering suffers from observational equivalence, whereby both faithful causal rationales and spurious post-hoc rationales can yield same outputs and are statistically indistinguishable from observation alone. Inspired by independent causal mechanisms, we argue that faithful rationales imply transfer invariance, in which causal rationales should remain effective across logically similar samples. We instantiate this criterion as Executable Causal Intervention for Demonstrations Distillation (ECI2D), which abstracts language rationales into executable programs and applies the causal interventions ( $do$ -operator) on logically similar samples, retaining only those demonstrations that imply transfer invariance. Experiments on math, logical and commonsense reasoning benchmarks demonstrate that ECI2D consistently outperforms observation-based ICL while retaining far fewer demonstrations, with gains exceeding 7 points as the test distribution shifts.

Figures & tables

Figure 1: Illustration of post-hoc rationalization using Llama3-8B on the GSM8K Cobbe et al. (2021) . Early Answering denotes the original CoT is truncated (denoted by gray ) to test if the answer is predetermined, and Adding Mistakes denotes a logical error is injected (highlighted in red ) before answering the question.

Panel of Figure 2 — Figure 2: Structural Causal Models (SCMs) in LLM inference. (a) Observed SCM: It treats the observed rationale as the direct causal mediator, forming a linear chain $X\to R_{o}\to Y$ . (b) True SCM: The actual mechanism comprises a faithful causal path $X\to R_{c}\to Y$ and a spurious post-hoc path where $R_{s}$ is dependent on both $X$ and $Y$ ( $X\dashrightarrow R_{s}\dashleftarrow Y$ ). The two SCMs are observational equivalence as discussed in Section 3 .

Figure 3: Invariant Reasoning Distillation phase in ECI2D. ECI2D distills faithful causal demonstrations through an Abstract-Intervene-Verify pipeline. Step 1 : Causal Abstraction. For each sample $(x_{i},y_{i})$ , the LLM generator $\Theta_{\text{Gen}}$ produces a natural language rationale $\hat{r}_{i}$ , which is then abstracted by LLM Abstractor $\Theta_{\Pi}$ into an executable program $\hat{p}_{i}$ . Step 2 : Executable Causal Intervention. For all logically similar samples $(x_{j},y_{j})\in\mathcal{N}(x_{i})$ , LLM adapter $\Theta_{\text{Adapter}}$ transplants the logical skeleton of $\hat{p}_{i}$ to the logically similar inputs to generate $\hat{p}_{i\to j}$ , performing an executable intervention $do(R=\hat{r}_{i})$ on $(x_{j},y_{j})$ via execution. Step 3 : Transfer Invariance Verification. The intervention output is scored against the ground truth $y_{j}$ . Rationales that fail to generalize are identified as spurious and discarded, while those demonstrating transfer invariance are retained in the causal pool $D_{\text{causal}}$ .

Method	Math Reasoning						Logical Reasoning
Method	GSM8K	SVAMP	GSM-S	P1	P2	Ret.	DROP	Date U.	Obj. C.	Word S.	Ret.
Meta-Llama-3-8B-Instruct
Direct	6.97	63.50	70.12	55.46	31.96	–	62.69	42.00	40.40	53.60	–
Zero-shot CoT	71.95	80.10	67.16	54.06	29.76	–	74.94	70.80	73.20	52.80	–
Manual Few-shot CoT	79.00	81.30	75.32	58.54	32.04	–	76.47	73.20	86.00	54.00	–
Self-Gen. Reinforced ICL	81.88	84.80	76.88	63.74	36.92	86%	74.99	53.60	77.60	50.80	63%

Table 1: Performance comparison on math and logical reasoning benchmarks. Following standard CoT Wei et al. (2022) , we use 8-shot demonstrations for math and 3-shot for logical tasks. We report accuracy (%) for all benchmarks except DROP (F1 %). + Code denotes the self-generated reinforced ICL with program-augmented demonstrations, which is the same rationale-and-program format as ECI2D. Retention rates after distillation are included. Best results per benchmark are bold , second-best are underlined . GSM-S = GSM-Symbolic, P1/P2 = Plus-1/Plus-2, Date U. = Date Understanding, Obj. C. = Object Counting, Word S. = Word Sorting, Ret. = Retention

Benchmark	Llama3-8B			Qwen2.5-7B
Benchmark	Prog.	Lang.	Both	Prog.	Lang.	Both
GSM8K	74.45	80.89	82.11	85.14	91.58	91.74
SVAMP	85.70	86.10	86.60	92.20	93.60	95.50
GSM-S	74.98	76.04	77.36	85.18	89.64	89.80
GSM-S P1	62.22	64.56	66.02	76.92	81.64	82.02
GSM-S P2	36.96	37.12	37.72	60.20	64.60	65.28

Table 2: Ablation study analyzing different rationale components. We compare performance across three configurations: Program-only

\hat{p}

(Prog.), Language rationale

\hat{r}

alone (Lang.), and Both (

\hat{p}

and

\hat{r}

). Best per model and benchmark in bold . GSM-S = GSM-Symbolic, P1/P2 = Plus-1/Plus-2.

Model	$N$	Num	GSM8K	SVAMP	GSM-Symbolic
Model	$N$	Num	GSM8K	SVAMP	Base	P1	P2
Llama	3	3913	82.11	86.60	77.36	66.02	37.72
	5	2248	79.61	87.20	78.38	64.28	37.80
	8	1387	79.30	87.10	77.86	64.76	37.40
Qwen	3	4239	91.74	95.50	89.80	82.02	65.28
	5	2417	87.79	89.70	83.46	73.74	56.32

Table 3: Impact of sample number

N

on performance. We report accuracy (%) across benchmarks. The column ‘ Num ’ denotes the size of the causal pool

|D_{\text{causal}}|

Config	Abs. (%)	Adpt. (%)	Ret. (%)	Acc (%)
Standard	1.5	3.5	57.3	91.74
W/o constraints	6.3	12.2	41.3	88.95
Qwen2.5-1.5B	10.6	25.1	24.6	87.48

Table 4: Impact of Abstractor(Abs.)/Adapter(Adpt.) degradation on error propagation and downstream performance (GSM8K, Qwen2.5-7B). Ret. = Retention

Method	CSQA	Sports	Ruin	Ret.
Direct	79.93	72.40	50.80	–
Zero-shot CoT	79.03	74.40	53.60	–
Manual Few-shot CoT	79.52	86.80	64.00	–
Self-Gen. Reinforced ICL	80.34	74.80	43.60	79%
+ Code	78.73	72.80	46.40	77%
ECI2D-Hybrid (ours)	82.64	79.20	47.20	59%

Table 5: Performance of the hybrid abstraction. The pool is distilled on CommonsenseQA alone, so the two BBH subsets measure cross-domain transfer. Following standard CoT Wei et al. (2022) , we use 7-shot demonstrations on CommonsenseQA and 3-shot on BBH, and report accuracy (%). + Code restricts the abstraction to a deterministic Python program and retains candidates by observational correctness alone. Best per benchmark in bold , second-best underlined . Sports = Sports Understanding, Ruin = Ruin Names, Ret. = Retention.

Method	GSM8K	SVAMP	GSM-Symbolic	GSM-Symbolic-P1	GSM-Symbolic-P2	Retention
Direct	42.30	58.80	72.74	69.76	60.20	–
Zero-shot CoT	80.26	81.60	74.78	72.84	66.08	–
Manual Few-shot CoT	82.26	84.80	82.70	80.08	72.72	–
Self-Gen. Reinforced ICL	88.44	88.40	89.30	84.94	78.24	91%
ECI2D (Ours)	92.04	92.40	91.94	89.32	79.72	53%

Table 6: More results on math reasoning benchmarks using DeepSeek-R1-Distill-Qwen-7B . We utilize 8-shot demonstrations. We reproduce all baselines and report accuracy across all benchmarks. The demonstration retention rate after distillation is also reported. The best results for each setting are marked in bold .

Method	GSM8K	SVAMP	GSM-Symbolic	GSM-Symbolic-P1	GSM-Symbolic-P2	Retention Rate
Direct	73.09	62.60	74.28	71.96	64.80	–
Zero-shot CoT	87.88	80.00	85.62	76.88	70.16	–
Manual Few-shot CoT	84.42	74.60	79.14	82.76	72.48	–
Self-Gen. Reinforced ICL	89.95	89.60	90.36	83.40	80.52	88%
ECI2D (Ours)	94.54	93.20	93.80	88.12	82.24	54%

Table 7: More results on math reasoning benchmarks using Qwen3-8B . Following Team (2025) , we utilize 4-shot demonstrations. We reproduce all baselines and report accuracy across all benchmarks. The demonstration retention rate after distillation is also reported. The best results for each setting are marked in bold .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Boolean Expressions	Penguins in a Table
Llama3	Manual Few-shot CoT	72.40	47.15
	Self-Gen. Reinforced ICL	68.40	44.52
	ECI2D (Ours)	76.00	47.26
Qwen2.5	Manual Few-shot CoT	83.60	47.58
	Self-Gen. Reinforced ICL	87.20	46.58
	ECI2D (Ours)	91.60	47.13

Appendix

Table 8: More results on logical reasoning benchmarks across different models. The best results for each setting are marked in bold , and the second-best results are underlined .

Benchmark	Llama3-8B		Qwen2.5-7B
Benchmark	Seed 42	3 seeds	Seed 42	3 seeds
GSM8K	82.11	82.36 $\pm$ 0.23	91.74	91.96 $\pm$ 0.23
SVAMP	86.60	86.70 $\pm$ 0.36	95.50	95.53 $\pm$ 0.65
GSM-S	77.36	77.35 $\pm$ 0.13	89.80	89.79 $\pm$ 0.12
GSM-S P1	66.02	66.05 $\pm$ 0.04	82.02	82.08 $\pm$ 0.18
GSM-S P2	37.72	37.88 $\pm$ 0.16	65.28	65.40 $\pm$ 0.14

Appendix

Table 9: Stability of ECI2D across three independent distillation seeds (

42

43

44

), with the causal pool

D_{\text{causal}}

rebuilt from scratch for each seed. Seed 42 reproduces the run reported in Table 1 . Ret. = Retention.

Method	GSM8K	GSM-S P2	DROP
Self-Gen. Reinforced ICL	81.88	36.92	74.99
@ matched pool size	79.15	35.24	73.54
ECI2D (ours)	82.11	37.72	80.87

Appendix

Table 10: Size-matched control with Meta-Llama-3-8B-Instruct . The observational pool is subsampled from

6{,}427

3{,}913

candidates for math reasoning and from

2{,}981

1{,}278

for DROP, matching

|D_{\text{causal}}|

in each domain. Retrieval, shot count, prompt format and decoding are identical throughout.

Method	GSM8K	SVAMP	GSM-S	P1	P2
Manual Few-shot CoT	79.00	81.30	75.32	58.54	32.04
Self-Gen. Reinforced ICL	81.88	84.80	76.88	63.74	36.92
PoT †	81.65	85.20	75.44	58.62	32.28
ECI2D (ours)	82.11	86.60	77.36	66.02	37.72

Appendix

Table 11: Comparison with program-aided prompting on math reasoning ( Meta-Llama-3-8B-Instruct ). PoT † generates and executes a program for each test query and therefore requires interpreter access at inference time; the remaining methods issue a standard few-shot prompt. No ranking is marked, since the two regimes operate under different deployment assumptions. GSM-S = GSM-Symbolic, P1/P2 = Plus-1/Plus-2.

Cause	Share (%)
Wrong option returned	94.92
Execution timeout	3.57
Code error	1.51

Appendix

Table 12: Decomposition of the interventions scoring

S_{\text{inv}}=0

under the hybrid abstraction on CommonsenseQA.

Target	Source	Samples	Agreement
neighborhood membership	GSM8K	1,000	94.4 (93.6–95.3)
	DROP	1,000	91.3 (90.3–91.8)
$\langle\hat{r},\hat{p}\rangle$ alignment	both	2,000	95.5 (94.5–96.5)
Chain correctness in $D_{\text{causal}}$	GSM8K	500	100.0 [99.24, 100]

Appendix

Table 13: Human validation of the assumptions underlying ECI2D. Three annotators judged every sample independently and blind to condition. Ranges give the per-annotator minimum and maximum.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 402 papers

ACIL: Auto Chain of Thoughts for In-Context Learning
May 16, 2026Rui ChuLLM Prompting CoT Reasoning
Many-Shot CoT-ICL: Making In-Context Learning Truly Learn
May 13, 2026Tsz Ting Chung, Lemao Liu, Mo Yu +1CoT Reasoning In-Context Learning
Hong Kong University of Science and Technology · Fudan University · Wechat AI, Tencent.
When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning
May 25, 2026Chenghao Qiu, Chunli Peng, Yufeng Yang +2In-Context Learning
Texas A&M University

May 16, 2026J/K move · Enter open · S save

ACIL: Auto Chain of Thoughts for In-Context Learning

Rui Chu

LLM Prompting CoT Reasoning In-Context Learning

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have shown that Chain-of-Thought (CoT) reasoning can substantially improve performance on complex reasoning tasks. At the same time, In-Context Learning (ICL) has become an important mechanism for adapting LLMs to new tasks without updating model parameters, using only examples provided in the prompt. However, standard ICL often struggles on tasks that require multi-step reasoning, because the demonstrations usually contain only input-output pairs and lack explicit intermediate reasoning steps. This paper introduces an Automatic Chain-of-Thought (Auto-CoT) framework to improve ICL by automatically constructing reasoning-enhanced demonstrations. Auto-CoT generates reasoning chains for input-output examples, augments the prompt context with structured intermediate explanations, and removes irrelevant or low-quality demonstrations through a systematic selection process. By incorporating high-quality reasoning examples into the ICL prompt, Auto-CoT guides the model toward more reliable reasoning and improves prediction accuracy. Experiments across multiple reasoning tasks demonstrate that the proposed framework improves ICL performance by providing explicit intermediate reasoning guidance.

Abstract

In-Context Learning

Explore similar work

ACIL: Auto Chain of Thoughts for In-Context Learning

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning