PragAlign: Evidence-Sensitive Reply Assistance Across Chinese and Japanese Appropriateness Judgments

Xin Zhong; Satori Hachisuka

cs.CLAug 27, 2026

PragAlign: Evidence-Sensitive Reply Assistance Across Chinese and Japanese Appropriateness Judgments

Authors: Xin Zhong, Satori Hachisuka

Organizations: The University of Tokyo

Abstract

Reply assistance in multilingual settings requires linguistic competence and culturally situated judgments of appropriateness. We present PragAlign, which separates context reading from selective clarification, and evaluate it alongside Direct and Rule. Nine native Chinese speakers judged Chinese materials, while three native Japanese speakers judged matched Japanese versions. In the Chinese evaluation, PragAlign received significantly better ranks than both baselines. In the Japanese evaluation, Direct had the lowest mean rank, PragAlign had the highest top-rank rate, and the omnibus difference was not significant. The groups selected the same top condition in 5 of 10 scenarios, including four shared PragAlign selections. The results identify shared and language-specific judgment patterns and inform reply assistance designed to support linguistic and cultural understanding.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs
Aug 14, 2026Florian BraunPragmatic Reasoning in Language Models Cross-Lingual Reasoning in Language Models
Independent Researcher
Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese
Aug 3, 2026Naho Orita, Hayato Ogawa, Daisuke KawaharaLLM Auditing Social Bias in Language Models
Waseda University
Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models
Oct 20, 2026Haonan Huang, Yuwei Lu, Chengkun CaiMultilingual Language Model Evaluation Cross-Lingual Consistency in Language Models
Princeton University · Independent Researcher · University College London

Aug 14, 2026J/K move · Enter open · S save

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs

Florian Braun

Independent Researcher

Pragmatic Reasoning in Language Models Cross-Lingual Reasoning in Language Models Multilingual Language Model Evaluation Language Model Probing Activation Steering

Open paper AskPDF arXiv

Large language models work well on English and behave in poorly understood ways on languages typologically far from it. Japanese is a clean example, where evaluation still leans on translation quality and JGLUE-style benchmarks, which roll lexical, syntactic and pragmatic competence into a single score. The phenomena on which general-purpose models fail Japanese users are pragmatic: honorifics, in-group and out-group reference, context-sensitive politeness, zero anaphora. I introduce J-PragEval-v0, a minimal-pair benchmark isolating four such phenomena from surface fluency, and combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B (28 layers, hidden size 1536) the corresponding contrasts live. The four features split three ways. Honorific register sits cleanly in the residual stream: 0.96 balanced accuracy at layer 15, and the model flips its preferred continuation with the scenario on 93 percent of items. Implicit subject and in-group reference are not linearly decodable at the final prompt token (0.48 and 0.38), yet flip rates are 0.77 and 0.79, so the contrast is worked out during generation rather than stored at the prompt. Indirect refusal is the negative case: 0.95 probe accuracy collapsing to a 0.43 flip rate under length-normalised teacher forcing, because the current minimal pairs conflate politeness with continuation length. I also specify Pragmatic Representation Steering, a parameter-free inference-time method that edits residual-stream activations along the class-mean-difference directions probing identifies. Feasibility is argued indirectly rather than demonstrated: the contrastive activation addition baseline, the same geometry the method would inject, recovers probe accuracy within one to two points of logistic regression wherever a linear signal exists. Scaling to Llama-3.1-Swallow-8B is the next step.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs

Human-LLM Alignment in Language Attitudes Toward Non-Native Japanese

Same Judgment, Different Yes: Measuring Multilingual Judgment Profiles in Language Models