Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval

Jianzhi Yan; Zhiming Li; Le Liu; Zike Yuan; Shiwei Chen; Youcheng Pan; Buzhou Tang; Yang Xiang

cs.AIApr 7, 2026

Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval

Authors: Jianzhi Yan, Zhiming Li, Le Liu, Zike Yuan, Shiwei Chen, Youcheng Pan, Buzhou Tang, Yang Xiang,

Organizations: Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China

Abstract

Large language models (LLMs) have made notable progress in logical reasoning, yet still fall short of human-level performance. Current boosting strategies rely on expert-crafted in-domain demonstrations, limiting their applicability in expertise-scarce domains, such as specialized mathematical reasoning, formal logic, or legal analysis. In this work, we demonstrate the feasibility of leveraging cross-domain demonstrating examples to boost the LLMs' reasoning performance. Despite substantial domain differences, many reusable implicit logical structures are shared across domains. In order to effectively retrieve cross-domain examples for unseen domains under investigation, in this work, we further propose an effective retrieval method, called domain-invariant neurons-based retrieval (DIN-Retrieval). Concisely, DIN-Retrieval first summarizes a hidden representation that is universal across different domains. Then, during the inference stage, we use the DIN vector to retrieve structurally compatible cross-domain demonstrations for the in-context learning. Experimental results in multiple settings for the transfer of mathematical and logical reasoning demonstrate that our method achieves an average improvement of 1.8 over the state-of-the-art methods \footnote{Our implementation is available at https://github.com/Leon221220/DIN-Retrieval}.

Figures & tables

Figure 2: Overview of the proposed DIN-based ICL framework. The model identifies domain-invariant neurons (DINs) from source and target activations (A) , constructs a stable DIN vector for representation (B) , retrieves demonstrations via DIN vector similarity (C) , and performs cross-domain chain-of-thought reasoning (D) .

	$\displaystyle\mathcal{I}=$	$\displaystyle\left\{k\in[1,d]\mid z_{k}^{S}>\tau\wedge z_{k}^{T}>\tau\right\}$
		$\displaystyle\cup\ \left\{k\in[1,d]\mid z_{k}^{S}<-\tau\wedge z_{k}^{T}<-\tau\right\}$

Algorithm 1 DIN-Retrieval

Model	DIN	Random	$\Delta$	Significant
LLaMA3.1-8B	$\textbf{62.7}_{\pm 2.3}$	$60.3_{\pm 1.8}$	+2.4	$\blacktriangledown$
Qwen2.5-7B	$\textbf{62.8}_{\pm 2.0}$	$59.5_{\pm 1.6}$	+3.3	$\blacktriangledown$
Qwen2.5-14B	$\textbf{70.7}_{\pm 1.2}$	$68.8_{\pm 1.2}$	+1.8	$\blacktriangledown$
Qwen3-8B	$\textbf{85.5}_{\pm 0.8}$	$84.0_{\pm 1.9}$	+1.5	$\blacktriangledown$

Table 1: Comparison of cross-domain reasoning (GSM8K

\to

FOLIO) accuracy between DIN-ICL and random neuron selection across different models .

\Delta

denotes accuracy gain, and

\blacktriangledown

indicates statistically significant improvement (

p<0.05

Method	Model Series	Parameters	Source Domain $\rightarrow$ Target Domain
Method	Model Series	Parameters	FOL $\rightarrow$ GSM	GSM $\rightarrow$ FOL	GSM $\rightarrow$ PRO	PRO $\rightarrow$ GSM	Average
Zero-shot	Qwen-2.5	7B	91.1	61.8	95.6	89.9	84.6
		14B	93.4	67.4	91.8	94.3	84.2
		32B	91.7	70.3	99.8	91.7	87.3
	Qwen-3	8B	93.3	81.7	100.0	92.4	91.8
		14B	89.9	84.5	96.2	90.5	90.2

Table 2: Performance (%) across cross-domain tasks using different ICL strategies. Each cell under DIN-ICL (Ours) includes a delta compared to the corresponding Zero-shot result. The final column reports average accuracy across tasks, where underlined bold denotes the best and italic denotes the second best. Significance testing was assessed via an unequal variances t-test in comparison with Zero-Shot :

\blacktriangledown

represents a p-value lower than 0.05.

Method	Model Series	Parameters	Source Domain $\rightarrow$ Target Domain
Method	Model Series	Parameters	FOL $\rightarrow$ GSM	GSM $\rightarrow$ FOL	GSM $\rightarrow$ PRO	PRO $\rightarrow$ GSM	Average
Set-BSR	Qwen-2.5	7B	89.6	59.7	95.4	89.2	83.5
		14B	92.9	67.4	94.2	93.2	84.8
		32B	91.6	66.0	99.6	91.6	85.7
	Qwen-3	8B	93.1	81.2	99.6	92.2	91.5
		14B	89.9	83.1	94.8	91.1	89.3

Table 3: Comparison between retrieval-based ICL (Set-BSR) and DIN-ICL.

Figure 3: Perplexity increase from pruning DINs vs. random neurons across the last six layers. Results are averaged over 300 trials. The solid line denotes mean PPL increase after pruning DINs, while the dashed line and shaded areas indicate the random pruning baseline (mean with 95th and 99th percentiles)

\blacktriangledown

indicates statistically significant improvement (

p<0.05

Figure 4: Case study illustrating a linear reasoning topology.

Figure 5: Case study illustrating a binary branching reasoning topology

Figure 6: Case study illustrating a blocking-condition reasoning topology.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Setting
Temperature	0.6
Sampling (Top-p/k)	Top-p = 1.0, Top-k = 50
Max Gen Length	8192 tokens
Random Seed	1-30

Appendix

Table 4: Decoding setup used throughout all experiments.

Figure 7: Effect of key hyperparameters on DIN-based ICL performance. Left: Increasing

k_{\mathrm{ratio}}

generally leads to slightly higher DIN accuracy. Right: DIN subspaces extracted from deeper layers tend to outperform shallower ones.

Figure 8: Mean DIN accuracy across different combinations of $k_{\mathrm{ratio}}$ and layer range.

Dataset	#Train	#Test	Labels
GSM8K	7,473	1,319	Numeric
FOLIO	5,216	500	A/B/C
PrOntoQA	22,000	2,000	A/B (Yes/No)

Appendix

Table 5: Statistics of datasets used in our cross-domain experiments.

Setting	Model Reasoning and Output
Context	A Japanese game company created the game The Legend of Zelda.All games on the Top 10 list are made by Japanese game companies.If a game sells more than one million copies, then it will be included in the Top 10 list.The Legend of Zelda sold more than one million copies.
Zero-shot	Since the Legend of Zelda sold more than one million copies, it should be included in the Top 10 list. However, it is stated that the Legend of Zelda is not on the Top 10 list. This creates a contradiction with the information that all games on the Top 10 list are made by Japanese game companies and the Legend of Zelda is made by a Japanese game company. Therefore, the statement "The Legend of Zelda is not on the Top 10 list" must be true. Final answer: A
	Final answer: A (True)
Demostration	Question: In the last 25 years, 60 $\%$ of Scrabble champions have been women and the rest have been men. There is one champion per year. Of the men, 40 $\%$ had a beard. How many men with beards have been Scrabble champion? Reasoning: 40 $\%$ of champions are men because 100 - 60 = <<100-60=40>>40 10 champions have been men because 25 x .4 = <<25.4=10>>10 4 champions have been men with beards because 10 x .4 = <<10.4=4>>4
DIN-ICL	Since the Legend of Zelda sold more than one million copies and all games on the Top 10 list are made by Japanese game companies, and the Legend of Zelda is made by a Japanese game company, it must be on the Top 10 list. Therefore, the statement "The Legend of Zelda is not on the Top 10 list" is false. Final answer: B
	Final answer: B (False)

Appendix

Table 6: Comparison between zero-shot reasoning and DIN-ICL reasoning on a cross-domain example. DIN-guided retrieval corrects the logical inconsistency made by zero-shot prompting.

Setting	Model Reasoning and Output
Context	People in this club who perform in school talent shows often attend and are very engaged with school events.People in this club either perform in school talent shows often or are inactive and disinterested community members.People in this club who chaperone high school dances are not students who attend the school.All people in this club who are inactive and disinterested members of their community chaperone high school dances.All young children and teenagers in this club who wish to further their academic careers and educational opportunities are students who attend the school. Bonnie is in this club and she either both attends and is very engaged with school events and is a student who attends the school or is not someone who both attends and is very engaged with school events and is not a student who attends the school.
Zero-shot	Let’s break down the information step by step:1. If Bonnie chaperones high school dances, then she is not a student who attends the school (from the third statement).2. If Bonnie does not chaperone high school dances, then she performs in school talent shows often (from the second statement).3. If Bonnie performs in school talent shows often, then she is very engaged with school events (from the first statement).4. If Bonnie is very engaged with school events, then she is a student who attends the school (from the first statement).5. From steps 1 and 4, we can conclude that Bonnie cannot be both a student who attends the school and chaperone high school dances.6. From steps 2 and 3, we can conclude that if Bonnie does not chaperone high school dances, then she is a student who attends the school.7. From steps 5 and 6, we can conclude that Bonnie is not a student who attends the school if she chaperones high school dances, and she is a student who attends the school if she does not chaperone high school dances.8. From the fourth statement, we can conclude that if Bonnie is a student who attends the school, then she is a young child or teenager who wishes to further her academic career and educational opportunities.9. From steps 7 and 8, we can conclude that Bonnie is a young child or teenager who wishes to further her academic career and educational opportunities if she does not chaperone high school dances.10. From the fifth statement, we can conclude that if Bonnie is an inactive and disinterested member of the community, then she chaperones high school dances.11. From steps 7 and 10, we can conclude that Bonnie is an inactive and disinterested member of the community if she chaperones high school dances.12. From steps 9 and 11, we can conclude that Bonnie is both a young child or teenager who wishes to further her academic career and educational opportunities and an inactive and disinterested member of the community if she does not chaperone high school dances.Therefore, the statement is true.Final answer: A
	Final answer: A (True)
Demostration	Question: Forty percent of the students have elected to learn from home during the pandemic. The remaining students are divided into two equal groups, only one of which is physically in school on any day. What percent of students are present in school? Reasoning: 100-40=<<100-40=60>>60 percent of students are remaining. 60/2=<<60/2=30>>30 percent are present in school.
DIN-ICL	Letś break down the information step by step:1. If Bonnie chaperones high school dances, then she is not a student who attends the school (from the context).2. If Bonnie does not chaperone high school dances, then she performs in school talent shows often (from the given statement).3. If Bonnie performs in school talent shows often, then she is very engaged with school events (from the context).4. If Bonnie is very engaged with school events, then she is a student who attends the school (from the context).5. From steps 1 and 4, we have a contradiction: Bonnie cannot both be a student who attends the school and not be a student who attends the school.6. Therefore, the given statement "If Bonnie either chaperones high school dances or, if she does not, she performs in school talent shows often" must be false, because it leads to a contradiction.Final answer: B
	Final answer: B (False)

Appendix

Table 7: Comparison between zero-shot reasoning and DIN-ICL reasoning on a cross-domain example. DIN-guided retrieval corrects the logical inconsistency made by zero-shot prompting.

Cross-Domain Transfer Learning

Cross-domain transfer learning reuses knowledge across datasets, populations, devices, or tasks when their distributions and feature spaces differ.

See all 203 papers

CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation
Apr 21, 2026Jianzhi Yan, Le Liu, Buzhou Tang +3Domain Adaptation Cross-Domain Transfer Learning
Harbin Institute of Technology, Shenzhen, 518100, China · Pengcheng Laboratory, Shenzhen, 518100, China
Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning
Apr 7, 2026Le Liu, Zhiming Li, Jianzhi Yan +7In-Context Learning Cross-Domain Transfer Learning
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China
How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
Oct 3, 2026Ziheng Cheng, Yixiao Huang, Hanlin Zhu +1RL for Language Model Reasoning Reinforcement Learning
University of California, Berkeley

Apr 21, 2026J/K move · Enter open · S save

CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation

Jianzhi Yan, Le Liu, Buzhou Tang, Yang Xiang+2

Harbin Institute of Technology, Shenzhen, 518100, China · Pengcheng Laboratory, Shenzhen, 518100, China

Domain Adaptation Cross-Domain Transfer Learning Domain Adaptation for LLMs LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) have achieved substantial advances in logical reasoning, yet they continue to lag behind human-level performance. In-context learning provides a viable solution that boosts the model's performance via prompting its input with expert-curated, in-domain exemplars. However, in many real-world, expertise-scarce domains, such as low-resource scientific disciplines, emerging biomedical subfields, or niche legal jurisdictions, such high-quality in-domain demonstrations are inherently limited or entirely unavailable, thereby constraining the general applicability of these approaches. To mitigate this limitation, recent efforts have explored the retrieval of cross-domain samples as surrogate in-context demonstrations. Nevertheless, the resulting gains remain modest. This is largely attributable to the pronounced domain shift between source and target distributions, which impedes the model's ability to effectively identify and exploit underlying shared structures or latent reasoning patterns. Consequently, when relying solely on raw textual prompting, LLMs struggle to abstract and transfer such cross-domain knowledge in a robust and systematic manner. To address these issues, we propose CoDA, which employs a lightweight adapter to directly intervene in the intermediate hidden states. By combining feature-based distillation of CoT-enriched reference representations with Maximum Mean Discrepancy (MMD) for kernelized distribution matching, our method aligns the latent reasoning representation of the source and target domains. Extensive experimental results on multiple logical reasoning tasks across various model families validate the efficacy of CoDA by significantly outperforming the previous state-of-the-art baselines by a large margin.

Abstract

Cross-Domain Transfer Learning

Explore similar work

CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation

Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning

How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws