How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

Zhaolu Kang; Yingjie He; Kehan Jiang; Leqi Zheng; Jiachen Qian; Qianyuan Zhang; Chunlei Meng; Yujie Feng

cs.CLJan 13, 2026

How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

Authors: Zhaolu Kang, Yingjie He, Kehan Jiang, Leqi Zheng, Jiachen Qian, Qianyuan Zhang, Chunlei Meng, Yujie Feng,

Organizations: Hunyuan, Tencent · Peking University · Tsinghua University · City University of Hong Kong ·

Abstract

Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is difficult to evaluate automatically because scrambled sentences often admit multiple valid reorderings. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including Semantic Accuracy, Logical Validity, Structural Consistency, Robustness, and Information Density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent gap between meaning-oriented generation and exact structural reconstruction, suggesting that structural robustness is not an automatic byproduct of semantic competence.

Figures & tables

Figure 1: OrderProbe example. Humans recover the canonical order using semantic anchors, while LLMs often generate correct meanings but fail exact reconstruction.

Benchmark	Year	Language	Target Unit	Perturbation Level	Core Task
General Robustness Benchmarks
RoTBench ( Ye et al., 2024 )	2024	En	Tool-use I/O	Real-world Noise	Tool-use Robustness (Tool Learning)
Instruct-Robust ( Li et al., 2024 )	2024	En	Instructions	External Injection	Instruction-Following Robustness
ReDial ( Lin et al., 2025 )	2025	En (Dialect)	Logic Problems	Dialectal Variation	Reasoning (Math/Logic)
Idiom & Figurative Language Benchmarks
MMFLD ( Lai et al., 2023 )	2023	Multi	Fig. Language	None (Static)	Detection / Class.

Table 1: Positioning of OrderProbe. Existing robustness benchmarks mainly test surface-level noise or instruction-level perturbations, while idiom benchmarks treat expressions as static retrieval or transformation targets. OrderProbe uniquely evaluates deterministic internal-order reconstruction with exact-match scoring.

Figure 2: OrderProbe construction pipeline. We collect raw four-character expressions from multiple sources, apply expert filtering to obtain canonical roots, build semantic references via dictionary grounding and verified augmentation, and generate the full non-identity permutation space as structurally perturbed evaluation inputs.

Figure 3: Overview of OrderProbe. The chart displays the distribution of script typologies and syntactic structures across the 3543 samples.

Category	Metric	Linguistic Definition & Rationale
Primary	Recovery Rate	Exact match of canonical order $x$ . Measures global structural integrity.
Diagnostics	$S_{Rob}$	Robustness score based on degradation under $S_{n}$ permutations (MDR/MDA).
	$S_{Acc}^{mean}$	Semantic accuracy via reference-meaning alignment using Cross-Encoders and BERTScore.
	$S_{Logic}$	NLI-based entailment check between model output $e$ and reference meaning.
	$S_{Cons}$	Stability across permutation variants, capturing invariance to internal shuffles.
	$S_{Info}$	Information density penalizing verbosity while rewarding high information-to-token ratio.

Table 2: Evaluation Metrics on OrderProbe. We prioritize Recovery Rate as the global structural indicator. Other diagnostic metrics characterize Robustness, Semantic Accuracy, Logical Validity, Structural Consistency, and Information Density.

Model Family	Setting	Diagnostic Component Metrics					Main Metric
Model Family	Setting	$S_{Acc}^{mean}$	$S_{Cons}$	$S_{Logic}$	$S_{Rob}$	$S_{Info}$	Recovery. (%)
Tier 1: High-Performance Reasoning Models
Qwen-3-14B	Zero-shot	0.493	0.591	0.672	0.621	0.579	31.062
Qwen-3-14B	+CoT	0.429	0.741	0.811	0.603	0.536	31.527 ( $\uparrow$ 0.465)
Qwen-3-VLThink	Zero-shot	0.520	0.539	0.712	0.645	0.520	30.928
Qwen-3-VLThink	+CoT	0.471	0.716	0.888	0.632	0.533	48.376 ( $\uparrow$ 17.448)

Table 3: Comparative Assessment of Zero Shot and Chain of Thought Prompting across 12 LLMs on OrderProbe. This table reports performance metrics encompassing semantic accuracy, consistency and structural recovery.

Figure 4: Performance Comparison across 23 Permutation Patterns. The chart illustrates recovery rates under baseline, few shot, and chain of thought settings, revealing the impact of anchor displacement on model stability.

	Component Metrics					Main Metric
Language	$S_{Acc}^{mean}$	$S_{Cons}$	$S_{Logic}$	$S_{Rob}$	$S_{Info}$	Recov. (%)
Simplified Chinese	0.4817	0.6445	0.6854	0.6193	0.4631	25.20
Traditional Chinese	0.4715	0.6184	0.6663	0.5998	0.4040	20.32
Japanese	0.4602	0.5897	0.5278	0.6048	0.5388	19.66
Korean	0.3990	0.7625	0.3014	0.5857	0.3906	5.72

Table 4: Comparison across script settings. Logographic scripts (Chinese/Japanese) yield substantially higher reconstruction accuracy than Korean Hangul, which serves as a phonogrammatic control condition. Component metrics are reported for diagnostic analysis alongside the primary Recovery score.

Figure 5: Performance by syntactic category. Parallel structures are easiest due to redundancy and symmetry cues, while asymmetric dependency patterns remain harder for reconstruction. Metrics are reported to diagnose whether failures arise from semantic gaps, instability, or structural sensitivity.

Figure 6: Degradation under perturbed 3-shot inputs. Left: per-metric drops

\Delta S_{acc}

\Delta S_{cons}

\Delta S_{logic}

\Delta S_{rob}

\Delta S_{info}

, computed as unperturbed minus perturbed 3-shot under matched inference settings. Right: recovery rate for perturbed 3-shot inputs. Models are ordered by recovery rate (higher is more robust).

Panel of Figure 7 — Figure 7: Case Study on Literal Interpretation. Qwen-3-8B produces a fluent but non-canonical literal interpretation, while DeepSeek-V3 correctly retrieves the idiomatic metaphor.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	CE Model	CE Agg	Embedding	Sim Func	BERTScore	NLI Model	NLI Agg	Weight Scheme	$\rho$	Effect Dir.
Original	BGE-base	max	BGE-small / MiniLM	cosine	✓	mDeBERTa	bidir-max	fixed	—	+
V1	MiniLM CE	max	same	cosine	✓	mDeBERTa	bidir-max	fixed	0.917	+
V2	BGE-base	mean	same	cosine	✓	mDeBERTa	bidir-max	fixed	0.904	+
V3	BGE-base	max	distiluse	cosine	✓	mDeBERTa	bidir-max	fixed	0.873	+
V4	BGE-base	max	same	dot	✓	mDeBERTa	bidir-max	fixed	0.898	+
V5	BGE-base	max	same	cosine	✗	mDeBERTa	bidir-max	fixed	0.931	+

Appendix

Table 5: Metric stress tests under controlled substitutions. We vary the cross-encoder backbone, aggregation strategy, embedding model, similarity function, BERTScore inclusion, NLI backbone, NLI aggregation, and weight scheme.

\rho

denotes Spearman correlation of model rankings with the original pipeline. The qualitative effect direction remains unchanged across all tested variants.

Language	$N$	$S_{Acc}^{mean}$		$S_{Logic}$		$S_{Info}$
		$\rho$	MAE	$\rho$	MAE	$\rho$	MAE
ZH-CN	290	0.787	0.104	0.822	0.088	0.743	0.112
ZH-TW	430	0.754	0.112	0.801	0.096	0.717	0.124
JA	170	0.694	0.132	0.739	0.116	0.665	0.143
KO	110	0.723	0.126	0.762	0.109	0.685	0.138
Macro-average	–	0.740	0.119	0.781	0.102	0.703	0.129

Appendix

Table 6: Human calibration of diagnostic metrics.

\rho

denotes instance-level Spearman correlation between automated metric scores and mean human ratings, while MAE measures absolute score disagreement.

Language	Evaluation Method	$s^{mean}_{Acc}$	$s_{Cons}$	$s_{Logic}$	$s_{Rob}$	$s_{Info}$
ZH-CN	Human Audit	0.412	0.758	0.829	0.715	0.456
	Automated Metric	0.482	0.663	0.749	0.625	0.516
ZH-TW	Human Audit	0.405	0.731	0.779	0.701	0.336
	Automated Metric	0.480	0.641	0.699	0.613	0.406
JA	Human Audit	0.413	0.654	0.632	0.708	0.547
	Automated Metric	0.479	0.559	0.552	0.621	0.617

Appendix

Table 7: Comparison between human expert audits and automated multi dimensional evaluation scores across languages.

Controlled Output Type	$N$	$S_{Acc}^{mean}$	$S_{Logic}$	$S_{Info}$
Correct concise	400	0.824	0.887	0.748
Correct but verbose	400	0.807	0.872	0.493
Wrong structure, correct meaning	400	0.781	0.846	0.704
Contradictory explanation	400	0.366	0.185	0.561
Literal interpretation	400	0.438	0.318	0.605
Unrelated hallucination	400	0.174	0.121	0.318

Appendix

Table 8: Controlled sanity check of diagnostic metrics. Each controlled output type isolates a targeted failure pattern. Semantic and logical metrics remain high when only structural recovery is incorrect, whereas contradiction and hallucination substantially reduce the corresponding diagnostic scores. Verbosity primarily affects Information Density.

Item	Specification
Evaluation period	October 2025–January 2026
Decoding	Temperature 0 when exposed; otherwise provider/checkpoint defaults
Tuning	No task-specific tuning, reranking, or post-processing
Prompting	Exact zero-shot, CoT, and few-shot templates are provided in Appendix D
Output schema	Recovered expression plus one-line explanation; parsing-only ablation is reported in Appendix C
Parsing	Normalize Unicode/script formatting, extract the unique four-unit candidate, and count missing, ambiguous, malformed, or unparsable outputs as incorrect

Appendix

Table 9: Reproducibility checklist for the OrderProbe evaluation.

Model	Avg. Tokens	Frag. Rate (%)	Recovery (%)
Qwen-3-14B	4.31	22.4	11.8
Qwen-3-VLThink	4.28	21.7	12.6
DeepSeek-V3.2	4.47	25.1	14.3
GPT-4o	4.82	31.6	8.9
Gemini-2.5-Flash	4.76	29.8	9.4
GLM-4VThink	5.12	38.3	6.7

Appendix

Table 10: Tokenizer fragmentation statistics for Korean Hangul syllable-block inputs. Avg. Tokens denotes the average number of model-specific subword tokens used to encode a four-block Korean expression. Frag. Rate denotes the percentage of expressions encoded with more than four tokens.

Language	Items	Inputs	Human Recovery (%)
ZH-CN	120	360	88.6
ZH-TW	120	360	84.1
JA	100	300	72.8
KO	100	300	46.5

Appendix

Table 11: Human reconstruction baseline. Human participants recover the canonical expression from the same scrambled inputs used for model evaluation.

Language	Recovery (%)	Time (s)	Confidence
ZH-CN	88.6	7.4	4.32
ZH-TW	84.1	8.1	4.18
JA	72.8	10.6	3.76
KO	46.5	14.9	3.08

Appendix

Table 12: Human reconstruction difficulty statistics. Time denotes average completion time per input in seconds. Confidence is self-reported on a 1–5 scale.

Model	Setting	Recovery (%)	Parse Fail. (%)
DeepSeek-V3.2	+CoT	62.24 [60.71, 63.79]	0.9
Qwen-3-VLThink	+CoT	48.38 [46.72, 50.03]	1.4
Gemini-2.5-Flash	+CoT	41.56 [39.91, 43.18]	1.8
GPT-4o	+CoT	37.69 [36.08, 39.31]	2.2
Qwen-3-14B	Zero-shot	31.06 [29.58, 32.55]	1.1
Qwen-3-VLThink	Zero-shot	30.93 [29.47, 32.41]	1.3

Appendix

Table 13: Uncertainty estimates and parsing failure rates. Recovery is reported with bootstrap 95% confidence intervals computed by resampling canonical expressions rather than individual permutations. Parse Fail. denotes the percentage of outputs from which no valid recovered expression can be extracted.

Model	Zero-shot	CoT	Few-shot
Qwen-3-14B	31.06	31.53	52.40
Qwen-3-VLThink	30.93	48.38	61.20
Gemini-2.5-Flash	30.49	41.56	34.60
GPT-4o	23.25	37.69	35.30
GLM-4VThink	21.64	30.08	31.50
DeepSeek-V3.2	20.10	62.24	49.10

Appendix

Table 14: Recovery across prompting settings. Values are exact Recovery rates (%).

Setting	Recovery (%)	Bootstrap 95% CI
Zero-shot	20.98	[20.50, 21.50]
CoT	31.22	[30.60, 31.80]
Few-shot	36.80	[36.10, 37.50]

Appendix

Table 15: Aggregate uncertainty across prompting settings. Recovery is macro-averaged over model-level scores.

Script setting	Recovery (%)	Bootstrap 95% CI
ZH-CN	25.20	[24.31, 26.08]
ZH-TW	20.32	[19.57, 21.09]
JA	19.66	[18.63, 20.71]
KO	5.72	[5.09, 6.38]

Appendix

Table 16: Zero-shot Recovery by script setting. Values are macro-averaged over the 12 models.

Model	ZH-CN	ZH-TW	JA	KO
Qwen-3-14B	37.6	29.5	29.0	11.6
Qwen-3-VLThink	35.4	30.2	30.4	10.6
Gemini-2.5-Flash	35.3	29.4	28.1	10.2
GPT-4o	27.6	23.0	21.4	6.8
GLM-4VThink	26.1	21.3	20.9	5.6
DeepSeek-V3.2	24.9	19.2	19.4	4.8

Appendix

Table 17: Zero-shot Recovery by model and script setting (%). The script-level pattern is broadly preserved across model families.

Frequency Bin	Recovery (%)
High	30.576
Mid	19.794
Low	10.231

Appendix

Table 18: Pilot Frequency-Stratified Recovery (Zero-shot). Exact recovery rates (%) across high-, mid-, and low-frequency bins.

Freq. Bin	Full Signal	Reduced Signal	Minimal Signal
High	61.25	19.74	8.32
Mid	39.74	10.62	6.83
Low	18.89	7.26	3.85

Appendix

Table 19: Frequency-stratified reconstruction accuracy under input-signal ablation. Exact recovery rates (%) are reported across high-, mid-, and low-frequency bins as semantic support in the input and prompt is progressively reduced.

Language	High Freq.	Mid Freq.	Low Freq.
ZH-CN	36.8	25.4	14.7
ZH-TW	31.2	20.7	11.6
JA	29.5	19.8	10.9
KO	10.8	5.6	2.4
Average	30.2	19.5	9.9

Appendix

Table 20: Frequency-stratified recovery by script setting. Exact recovery rates (%) are reported across high-, mid-, and low-frequency bins constructed within each language.

Predictor	Coef.	Std. Err.	$p$ -value
$\log$ Frequency	0.42	0.06	$<.001$
ZH-TW vs. ZH-CN	-0.31	0.08	$<.001$
JA vs. ZH-CN	-0.37	0.09	$<.001$
KO vs. ZH-CN	-1.84	0.12	$<.001$
Parallel structure	0.28	0.07	$<.001$
Verb–Object structure	-0.19	0.06	.002

Appendix

Table 21: Logistic regression analysis of exact recovery. The dependent variable is whether the canonical expression is exactly recovered. Predictors include expression frequency, script setting, and syntactic structure.

Predictor set	Pseudo- $R^{2}$	Root-grouped AUC
Lexical familiarity	0.052	0.615
Structural configuration	0.109	0.688
Familiarity and structure	0.162	0.735
Full specification	0.281	0.812

Appendix

Table 22: Incremental explanatory power of lexical familiarity and structural factors.

Canonical group	Pairs	0 adj.	1 adj.	2 adj.
High-score	27,608	18.4	28.8	40.2
Low-score	14,908	4.8	7.6	13.1
Difference	—	13.6	21.2	27.1

Appendix

Table 23: Canonical-explanation probe. Recovery (%) on scrambled inputs by retained local adjacency.

Dataset	Exact (%)	Sem. Fid.
Idiomatic Expr.	30.4	0.81
Synthetic Expr.	16.8	0.63

Appendix

Table 24: Reconstruction performance on idiomatic expressions and synthetic non-idiomatic expressions under the same permutation protocol.

Output schema	Inputs	Recovery (%)
Standard expression + explanation	Full benchmark	20.98
Reconstruction only	200 stratified inputs	43.50

Appendix

Table 25: Reconstruction-only output ablation. The two values use different evaluation scopes and are therefore indicative comparisons.

Category	Definition	Rate (%)
Exact canonical recovery	Extracted expression matches the gold canonical form	20.98
Related but non-canonical	Another well-formed expression shares at least two target characters or the core meaning	12.80
Valid but unrelated	Well-formed expression without target overlap or core meaning	8.70
Literal recombination	Coherent compositional reading without recovered expression	31.50
Malformed / invalid	No unique well-formed expression or classifiable explanation	4.60
Other incorrect	Incorrect output not covered by the preceding categories	21.42

Appendix

Table 26: Estimated error categories under zero-shot prompting. Categories are mutually exclusive.

Model Family	$S_{Acc}^{mean}$	$S_{Cons}$	$S_{Logic}$	$S_{Rob}$	$S_{Info}$	Recovery (%)
Tier 1: High-Performance Reasoning Models
Qwen-3-14B	0.862	0.857	0.873	0.904	0.841	100
Qwen-3-VLThink	0.884	0.853	0.886	0.913	0.832	100
Gemini-2.5-Flash	0.871	0.864	0.859	0.898	0.821	100
Tier 2: Mid-Range General Models
GPT-4o	0.879	0.882	0.858	0.914	0.843	100

Appendix

Table 27: Reference Diagnostic Metrics under Unperturbed Inputs. This appendix table reports diagnostic component scores obtained from a reference evaluation run using unperturbed inputs and the same inference configurations as the main experiments. Since the inputs are already in canonical order, exact reconstruction is trivially easier in this setting, and all evaluated models achieve perfect recovery on these inputs. These results are provided for contextual reference only and are not used in any comparative or statistical claims.


Language	Type	Baseline Prompt Template (Visualized Layout)

	System	你是一个严格遵循格式要求的语言专家。请你回答为中文。请只输出一行，不要多余空行或其他文字：判断含义：[解释]
ZH-CN	User	请你判断这个的含义。直接判断并说出含义，不要有其他废话，尽可能全面：判断含义：[解释] 成语：{idiom}
	System	你是一個嚴格遵循格式要求的語言專家。請用繁體中文回答。請只輸出一行，不要多餘空行或其他文字：判斷含義：[解釋]
ZH-TW	User	請你判斷這個的含義。直接判斷並說出含義，不要有其他贅詞，尽可能全面：判斷含義：[解釋] 成語：{idiom}

Appendix

Table 28: Visualized baseline prompt templates used for idiom meaning inference across four languages. Prompts are shown in their formatted layout to illustrate the structural constraints imposed on model outputs.


Language	Type	Few Shot Prompt Template Visualized Layout

ZH-CN	System	你是一个严谨的逻辑分析专家。请严格遵循用户的指令格式输出结果。禁止输出任何开场白或结束语。
	System	start deep thinking, please always use the thinking mode
	User	请仔细阅读以下文本分析示例，学习其分析逻辑和输出格式。然后，对最后给出的待分析文本进行同样的分析。示例 1 输入文本：惊人一鸣 Thinking Process：分析字面，惊人一鸣在语义上不通顺。观察字词组合，发现一鸣通常作为主语或状语，惊人作为结果。调整语序为一鸣惊人，符合一声鸣叫让人震惊的逻辑。 Final Meaning：比喻平时没有突出的表现，一下子做出惊人的成绩。示例 2 输入文本：添足画蛇 Thinking Process：分析字面，添足意味着增加脚，画蛇是动作主体。逻辑上先有画蛇，后有添足的多余行为。重组为画蛇添足。 Final Meaning：比喻做了多余的事，非但无益，反而不合适。示例 3 输入文本：求剑刻舟 Thinking Process：分析字面，求剑是目的，刻舟是手段。按照因果和动作发生的时间顺序，应先在船上刻记号，再去寻找剑。重组为刻舟求剑。 Final Meaning：比喻拘泥不知变通，不懂得根据实际情况处理问题。待分析文本输入文本：{idiom} 请严格模仿上述示例的逻辑和格式输出，必须严格两行。
ZH-TW	System	你是一個嚴謹的邏輯分析專家。請嚴格遵循用戶的指令格式輸出結果。禁止輸出任何開場白或結束語。

Appendix

Table 30: Visualized few shot prompt templates used for multilingual idiom analysis. Prompts are shown in formatted layout to illustrate in context reasoning patterns and the strict two line output constraint.


Language	Type	CoT Prompt Template Visualized Layout

ZH-CN	System	你是一个严谨的逻辑分析专家。请严格遵循用户的指令格式输出结果。禁止输出任何开场白或结束语。 start deep thinking, please always use the thinking mode
	User	请分析以下文本的深层含义。这是一个测试语言逻辑重组能力的任务。为了确保准确性，请务必一步步思考，严格遵循以下分析路径。 1 字义拆解分析文本中每个字的独立含义。 2 结构重组观察这些字之间可能存在的句法逻辑，判断是否存在语序错位。 3 语义推导基于重组后的逻辑，推导该文本的整体寓意。请严格按照以下特定格式输出，必须严格两行。 Thinking Process: 在这里写下你的详细分步推理过程，包括对字序和结构的分析。 Final Meaning: 基于推理得出的最终含义解释，只给结论，不要再解释推理过程，尽可能全面。输入文本：{idiom}
ZH-TW	System	你是一個嚴謹的邏輯分析專家。請嚴格遵循用戶的指令格式輸出結果。禁止輸出任何開場白或結束語。 start deep thinking, please always use the thinking mode
	User	請分析以下文本的深層含義。這是一個測試語言邏輯重組能力的任務。為了確保準確性，請務必一步步思考，嚴格遵循以下分析路徑。 1 字義拆解分析文本中每個字的獨立含義。 2 結構重組觀察這些字之間可能存在的句法邏輯，判斷是否存在語序錯位。 3 語義推導基於重組後的邏輯，推導該文本的實際整體寓意。請嚴格按照以下特定格式輸出，必須嚴格兩行。 Thinking Process: 在這裡寫下你的詳細分步推理過程，包括對字序和結構的分析。 Final Meaning: 基於推理得出的最終含義解釋，只給結論，不要再解釋推理過程，盡可能全面。輸入文本：{idiom}

Appendix

Table 31: Visualized CoT prompt templates used for multilingual idiom analysis. The prompts enforce a fixed two line output format and instruct stepwise reasoning for semantic and structural reconstruction.

Panel of Figure 8 — Figure 8: Baseline performance on Chinese scripts illustrating Simplified Chinese on the left and Traditional Chinese on the right. Even powerful models often fail to recover the canonical order.

Panel of Figure 10 — Figure 10: Comparison of prompting strategies on Simplified Chinese. Chain of Thought prompting elicits better reasoning but still suffers from hallucination during complex reordering.

Panel of Figure 12 — Figure 12: Comparison of prompting strategies on Japanese. This detailed analysis highlights the syntax recovery challenges in mixed script inputs.

Input Order Sensitivity in Language Models

See all 14 papers

Sep 3, 2026

Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations

Large Language Models (LLMs) demonstrate strong multilingual reasoning performance, yet their robustness to semantics-preserving structural variation remains underexplored, particularly for relatively free word-order languages. We investigate the structural sensitivity of multilingual LLMs using two linguistically grounded perturbation settings in Hindi and Malayalam: constrained constituent reordering and active-passive voice transformation. We introduce a benchmark dataset IndicReStruct, with two variants, GSM8K-Reordered and GSM8K-Voice, constructed from GSM8K while preserving semantic meaning. Across six state-of-the-art LLMs and multiple prompting strategies, we observe consistent and significant degradation in mathematical reasoning performance under structurally perturbed inputs. To further understand these failures, we perform qualitative error analysis and mechanistic interpretability experiments using residual-stream activation patching. Our analyses show that reasoning failures frequently arise from disruptions in entity-quantity alignment and that intermediate transformer layers contribute most strongly toward reasoning restoration. Overall, our findings suggest that current multilingual LLMs remain highly sensitive to surface syntactic realization and lack robust compositional invariance under structurally different but semantically equivalent inputs.

Karthika Nhayakkat, Rajat Verma, Maharaj Brahma +4

Department of CSE, Indian Institute of Technology Bombay · SCEE, Indian Institute of Technology Mandi · Department of CSE, Indian Institute of Technology Hyderabad

Numerical Reasoning in Language Models Multilingual Language Model Evaluation Language Model Robustness

Ask PDF

Oct 8, 2026

Structure Tax: How Structured Output affects LLMs Performance

Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent

structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from

whether to structure' to `how to structure' for optimal reasoning preservation.

Vineet Kumar, Kanishka, Bhuvanesh Mandora

PayPal Artificial Intelligence PayPal, Bengaluru India

Structured Output Generation Structured Reasoning in Language Models LLM Reliability

Ask PDF

Jun 24, 2026

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.

Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli

Stanford University

Multimodal Robustness LLM Auditing MLLM Evaluation Multimodal Model Evaluation Multimodal Understanding Input Order Sensitivity in Language Models

Ask PDF

Abstract

Input Order Sensitivity in Language Models

Explore similar work

Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations

Structure Tax: How Structured Output affects LLMs Performance

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models