Cross-Lingual Reasoning in Language Models

Follow this topic Ask about this topic

The field asks whether language models can understand evidence and reason reliably when questions, sources, and answers use different languages. Researchers test this with parallel and native-language benchmarks in mathematics, logic, multi-hop question answering, and culturally grounded tasks, while probing how script, language resources, and translation affect results. Common approaches include multilingual fine-tuning and reinforcement learning, translation and code-switching, cross-lingual distillation, and alignment of internal representations. Training can narrow language gaps, but English often remains advantageous, and gains can trade off with language fidelity, cultural grounding, or efficiency; no single reasoning language works best in every setting.

Latest papers

69 papers9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Cross-Lingual Reasoning in Language Models
New York University
ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection
Feb 25, 2026Changjiang Gao, Zixian Huang, Kaichen Yang +3RL for Language Model Reasoning Cross-Lingual Reasoning in Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, Jiangsu, China · Department of Linguistics and Translation, City University of Hong Kong, Hong Kong, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China +1
IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
Feb 18, 2026Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap +1Education LLM Evaluation
Department of Computer Sciencce and Engineering, Chhattisgarh Swami Vivekanand Technical University Bhilai, India
Unlocking Reasoning Capability on Machine Translation in Large Language Models
Feb 16, 2026Sara Rajaee, Sebastian Vincent, Alexandre Berard +3Structured Reasoning in Language Models Large Reasoning Models
University of Amsterdam · Cohere · Cohere Labs
Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamil
Feb 16, 2026Sukumar Kishanthan, Kumar Thushalika, Buddhi Jayasekara +1Numerical Reasoning in Language Models Cross-Lingual Reasoning in Language Models
Department of Computer Science and Engineering University of Moratuwa Moratuwa, Sri Lanka · Department of Electrical and Information Engineering University of Ruhuna Galle, Sri Lanka · Faculty of Information Technology and Communication Sciences Tampere University Tampere, Finland +1
Do Language Models Reason Across Languages?
Jan 10, 2026Yan Meng, Wafaa Mohammed, Christof MonzCoT Faithfulness Cross-Lingual Reasoning in Language Models
Language Technology Lab · University of Amsterdam
MultiZebraLogic: A Multilingual Logical Reasoning Benchmark
Nov 5, 2025Sofie Helene Bruun, Dan Saattrup SmartLogical Reasoning Multilingual Language Model Evaluation
The Alexandra Institute Rued Langgaards Vej 7, 5D, 2300 Copenhagen S, Denmark
How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?
Oct 2, 2025Wen Yang, Junhong Wu, Chong Li +2Cross-Lingual Reasoning in Language Models Cross-Lingual Transfer
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research
One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning
Sep 25, 2025Sualeha Farid, Jayden Lin, Zean Chen +2Multilingual Language Model Evaluation Moral Reasoning in Language Models
University of Michigan, Ann Arbor, USA

Feb 25, 2026J/K move · Enter open · S save

Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads

Shaswat Patel, Vishvesh Trivedi, Yue Han, Yihuai Hong+1

New York University

Multilingual Language Models Cross-Lingual Reasoning in Language Models Attention Mechanisms Multi-Head Attention Cross-Lingual Transfer Attention Head Specialization

Open paper AskPDF arXiv

Retrieval heads, a subset of attention heads in Transformers, were studied in English, showing its crucial role in retrieving information from the context. We expand the study of retrieval heads to multilingual context and find that while nearly half of all retrieval heads are often shared across multiple languages, language-specific retrieval heads also emerge. We further design a cross-lingual needle-in-the-haystack task, to identify

\textit{Retrieval Transition Heads (RTH)}

that retrieve key information and further map it to target-language output. Our experiments reveal that RTHs do not always overlap with RH, and can be more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuAD) and two model families (Qwen2.5-7B Instruct and Llama3.1-8B Instruct), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Specifically, for Llama3.1-8B Instruct, masking the top-25 RTHs results in an average 37.3-point drop (against 19.5-point drop from RH masking) in reasoning accuracy (MMLU-ProX, MGSM) and 9.0 F1-score reduction (against 5.1 drop for RH) in extractive QA (MLQA, XQuAD). Our work advances the understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.