Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs

Mohammad Reza Rezaei; Dingxian Wang; Nikhil Verma; Sajjad Pakdamansavoji; Andrew Rabinovich; Zhao Chen

cs.LGNov 1, 2026

Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs

Authors: Mohammad Reza Rezaei, Dingxian Wang, Nikhil Verma, Sajjad Pakdamansavoji, Andrew Rabinovich, Zhao Chen

Organizations: Upwork Inc.

Abstract

Large language models show large and seemingly arbitrary performance drops when a task is rewritten in ways that leave its requirements untouched. We recast task difficulty as a causal quantity: the treatment effect of meaning-preserving perturbations on a model. A meaning-preserving rewrite can only be observed to hurt a task the model can already solve, so we place the estimand on the principal stratum where the degradation occurs: the conditional causal difficulty (CCD), the probability that a randomly drawn twin breaks a solvable task. Because harder tasks tend to be more phrasing-sensitive, complexity enters as an effect modifier rather than a confounder, and we residualize it out to obtain complexity-residualized difficulty (CRD); zero for a task exactly as fragile as its complexity predicts, positive for anomalously fragile ones. Current benchmarks do not supply the data CCD requires, so we introduce GlassGSM and GlassMATH, augmenting GSM8K and MATH with twins built from perturbation families, raising fragility on GlassGSM by $\approx 12\times$ over GSM-IC-style distractors on the same problems ( $3.7\%\to44.9\%$ ) while keeping the answer key unchanged. Our predictor Frail learns the per-perturbation break probability and reaches $0.613{\pm}0.023$ of the suite reliability ceiling on GlassGSM and $0.557{\pm}0.023$ on GlassMATH, against $0.171$ / $0.401$ for a surface-feature baseline and $0.444$ / $0.463$ for supervised baselines such as VAP3, and, at a $5\%$ curation budget on GlassGSM, doubles random recall of fragile items, matching an oracle ranking. Additionally, cross-fitted residualization leaves CRD uncorrelated with the declared proxy in practice. Code is available at https://github.com/upwork/Conditional_Causal_Difficulty_for_LLMs

Figures & tables

Figure 1: Frail-

\pi

predicts CCD better than every baseline on both suites (left, middle; Spearman normalized by the reliability ceiling, mean

\pm

sd over five seeds), and remains predictive when the solver backbone that defines the stratum is changed (right; raw

\rho_{S}

of Frail-

\pi

trained and evaluated on each solver’s own CCD in the single-dose

m{=}9

run, so not on the same scale as the left panels). Details in Tables 2 and 4 .

Panel of Figure 2 — Figure 2: (a) A real pair from GlassGSM ( $m{=}1$ ): the twin $T(J)$ differs from $J$ only by moving the blue statement past an independent one. The model answers $J$ correctly but this twin incorrectly, so exact-match scoring gives $Y(J)=1.00$ and $Y(T(J))=0.00$ , an instance of $\Delta(J,T)$ with no requirement-level basis. (b) The graph shows why $C$ confounds an observational comparison across prompts through the back-door $S\leftarrow C\to Y$ (dashed), so any predictor of $Y(J)$ on a fixed phrasing estimates a complexity-inflated quantity. Our contrast does not open that back-door: $T$ is drawn independently of $C$ , so comparing $J$ against its own twin is identified.

Dataset	Problems	Twins	Fragile	Zero	R1	R2	R3
GSM-IC (on GSM8K test)	1,319	20	3.7%	90.1%	✓	$\times$	✓
GSM-Plus (MPPO) ( Li et al., 2024 )	1,319	2	10.5%	86.2%	✓	$\times$	✓
GSM-Symbolic p1	100	10	25.0%	56.0%	$\times$	✓	n/a
GSM-DC ( Yang et al., 2025 )	6,222	1	16.4%	64.0%	✓	✓	$\times$
GlassGSM (ours)	1,319	avg. 13.9	44.9%	48.6%	✓	✓	✓
MATH (original) ( Hendrycks et al., 2021 )	5,000	—	n/a	n/a	n/a	$\times$	n/a

Table 1: Where each real-math perturbation suite fails. Fragile :

d_{\mathrm{obs}}>0.05

, with

d_{\mathrm{obs}}

the measured UCD

D_{\mathrm{avg}}

over all problems (this table only; elsewhere, e.g. Appendix D.3 , a job is fragile when

\pi>0.05

on the solvable stratum); Zero :

|d_{\mathrm{obs}}|<0.001

. R1 No-Op (meaning preserving); R2 enough fragile problems to rank; R3 no inverse complexity–fragility confound. MATH (original) has no perturbation suite, so Fragile/Zero, R1, and R3 are n/a. GSM-IC ( Shi et al., 2023 ) releases 100 base problems; the GSM-IC row applies its irrelevant-sentence construction to all 1,319 GSM8K test problems (20 twins each), the base set GlassGSM also uses. For GSM-Plus, answer preservation was verified empirically (zero drift across the full test set). GlassGSM and GlassMATH numbers are for the released corpora pooled over all five doses (unique twins), resampled from the audited pools of Appendix D.4 .

Suite	Method	$\pi$ $\uparrow$	$a$ $\uparrow$	$b$ $\uparrow$
GlassGSM	Frail- $\pi$	0.613 ${\pm}$ 0.023	0.611 ${\pm}$ 0.020	0.289 ${\pm}$ 0.021
	VAP3	0.444 ${\pm}$ 0.049	0.441 ${\pm}$ 0.046	0.245 ${\pm}$ 0.031
	BERTPE	0.420 ${\pm}$ 0.051	0.420 ${\pm}$ 0.049	0.255 ${\pm}$ 0.035
	TF-IDF+SVD features, job-level regr.	0.320 ${\pm}$ 0.044	0.320 ${\pm}$ 0.049	0.184 ${\pm}$ 0.019
	Surface features (GBT)	0.171 ${\pm}$ 0.023	0.164 ${\pm}$ 0.026	0.146 ${\pm}$ 0.056
	Declared complexity score only	0.207 ${\pm}$ 0.060	0.192 ${\pm}$ 0.058	0.181 ${\pm}$ 0.049

Table 2: Test-fold Spearman normalized by the suite reliability ceiling (

0.969

for GlassGSM,

0.972

for GlassMATH; Appendix D.2 ), mean

{\pm}

sd over five seeds.

\pi

: CCD predicted by the single-tower head

g_{\omega}

;

a

b

: dose-response intercept/slope (Definition 2 ), from a logistic fit to the contrast head’s per-twin predictions across doses (uses the twins). Job-level regression : frozen TF-IDF+SVD features regressed directly on job-level

\pi

; it differs from Frail-

\pi

in both encoder and objective (§ 5.1 ). Frozen twin-distance / Prompt Clarity / Embedding Entropy / Embedding Specificity-IEF (from PromptSET, Razavi et al. 2025 ; Arabzadeh et al. 2020 ) are retargeted to

\pi

with a TF-IDF+SVD embedding on the same seeded splits (Appendix C.1 ); they have no per-dose grid, so

a

b

are n/a. The

a

b

columns use the same

\pi

ceiling as a common scale, not ceilings of their own.

Figure 3: Cross-model generalization, single-dose ( $m{=}9$ ) twins, mean $\pm$ sd over five seeds (sd ${<}0.005$ for bars without a visible error bar). Qwen and DeepSeek denote the Qwen3.7-plus and DeepSeek-v4-pro generators. The Frail- $\pi$ bars are the cross-generator run whose orthogonality diagnostics appear in Table 3 ; shown alongside two ablations not in that table: a frozen-encoder representation ( Frail- $\pi$ , frozen encoder ) and per-generator retrained BERTPE/VAP3. Other baselines follow Table 2 ’s definitions. Bars are raw (unnormalized) $\rho_{S}$ . This single-dose slice differs from the pooled 5-dose run behind Table 2 , so its numbers are not directly comparable to that table’s.

Suite / generator	MAE $\downarrow$	$\|\rho_{C}\|\downarrow$	target	pred.	pred.
			$\mathrm{Corr}(\cdot,C)$		$\mathrm{Corr}(\cdot,C^{2})$
GlassGSM, gpt-4o-mini twins	0.129	0.044	$-$ 0.043	$-$ 0.061	$-$ 0.068
GlassMATH, gpt-4o-mini twins	0.129	0.030	$0.046$	$-$ 0.023	$-$ 0.022
GlassGSM, Qwen3.7-plus twins	0.074	0.092	$-$ 0.058	$-$ 0.055	$-$ 0.069
GlassMATH, Qwen3.7-plus twins	0.117	0.067	$-$ 0.018	$-$ 0.016	$-$ 0.012
GlassGSM, DeepSeek-v4-pro twins	0.058	0.087	$-$ 0.025	$-$ 0.060	$-$ 0.056

Table 3: CRD orthogonality against

C

; the primary instantiation of Definition 3 . The declared proxy

C

is suite-specific: the MATH difficulty level (1–5) for GlassMATH (external, non-circular; § 4 ), and the number of reasoning steps in each problem’s GSM8K gold solution for GlassGSM, which ships no such label.

|\rho_{C}|

: absolute Spearman between the prediction and

C

;

\mathrm{Corr}

: Pearson. All rows are Frail-

\pi

on the single-dose (

m{=}9

) run of Figure 3 , differing only in the twin generator.

GlassGSM
train $\backslash$ eval	gpt-4o-mini	Qwen3.7-plus	DeepSeek-v4-pro
gpt-4o-mini	0.431	0.355	0.411
Qwen3.7-plus	0.297	0.299	0.401
DeepSeek-v4-pro	0.237	0.306	0.389
GlassMATH
train $\backslash$ eval	gpt-4o-mini	Qwen3.7-plus	DeepSeek-v4-pro

Table 4: Solver transfer. Held-out

\rho_{S}

of the fine-tuned Frail-

\pi

trained on the row solver’s measured

\pi

and evaluated against the column solver’s measured

\pi

, same items/twins throughout (

m{=}9

). Diagonal = in-distribution; off-diagonal = cross-solver transfer.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$\rho_{S}\uparrow$	MAE $\downarrow$	NDCG@10 $\uparrow$	AUROC $\uparrow$	$\|\rho_{C}\|$
Surface length	0.100	0.021	0.804	0.594	0.052
Score-predictor	0.496	0.014	0.829	0.727	0.052
Frozen twin-distance	$-$ 0.112	0.023	0.686	0.446	0.182
Frail- $\tau$ ex-ante (ours)	0.420	0.015	0.799	0.712	0.067
Frail- $\tau$ contrast (ours)	0.617	0.012	0.863	0.828	0.028

Appendix

Table 5: Synth-CD:

D_{\mathrm{avg}}

prediction (1,000 jobs,

G{=}12

twins, 300 held-out, MiniLM backbone). Text length is controlled to be independent of

C

and

E

. Score-predictor: predicts the base score

Y(J)

from text and uses it as a difficulty score.

\rho_{S}

, NDCG@10, AUROC

\uparrow

; MAE,

|\rho_{C}|

shown for context.

Suite	Representation	$\pi$ $\uparrow$	$a$ $\uparrow$	$b$ $\uparrow$
GlassGSM	TF-IDF+SVD	0.398 ${\pm}$ 0.042	0.395 ${\pm}$ 0.037	0.039 ${\pm}$ 0.070
	Frozen sentence encoder	0.381 ${\pm}$ 0.026	0.381 ${\pm}$ 0.032	$-$ 0.126 ${\pm}$ 0.112
	Frozen OpenAI embedding	0.326 ${\pm}$ 0.040	0.324 ${\pm}$ 0.039	$-$ 0.158 ${\pm}$ 0.046
GlassMATH	TF-IDF+SVD	0.435 ${\pm}$ 0.021	0.425 ${\pm}$ 0.021	0.334 ${\pm}$ 0.095
	Frozen sentence encoder	0.386 ${\pm}$ 0.013	0.394 ${\pm}$ 0.014	0.314 ${\pm}$ 0.037
	Frozen OpenAI embedding	0.431 ${\pm}$ 0.016	0.417 ${\pm}$ 0.015	0.369 ${\pm}$ 0.022

Appendix

Table 6: Representation ablation for Frail-

\pi

: test-fold Spearman normalized by the suite reliability ceiling (

0.969

for GlassGSM,

0.972

for GlassMATH; Appendix D.2 ), mean

{\pm}

sd over five seeds;

1.0

is the attainable maximum. Same protocol and normalization as Table 2 , so these rows are directly comparable to the main model’s

0.613

0.557

Lever	GlassGSM	GlassMATH
Main model (fine-tuned encoder)	0.613	0.557
Stronger backbone (RoBERTa-large)	0.543	0.548
Lower LR / higher patience	0.567	0.552
Larger job batch	0.606	—
Ensemble (mean-rank / mean-z)	0.559 / 0.537	0.549 / 0.551
Pooled training data (per-origin) §	0.536	0.548

Appendix

Table 7: Further training-recipe levers vs. the

0.613

0.557

main model (break rate

\rho_{S}

normalized by the suite reliability ceiling ,

0.969

0.972

; Appendix D.2 ; same normalization as Table 2 ,

1.0

is the attainable maximum; 5 seeds unless noted). None beats the main model meaningfully.

Method	$\rho_{S}\uparrow$	MAE $\downarrow$	NDCG@10 $\uparrow$	AUROC $\uparrow$	$\|\rho_{C}\|$
Surface length	0.162 ${\pm}$ 0.030	0.245 ${\pm}$ 0.009	0.453 ${\pm}$ 0.065	0.727 ${\pm}$ 0.041	0.086 ${\pm}$ 0.040
Correlational score-predictor	0.028 ${\pm}$ 0.036	0.254 ${\pm}$ 0.010	0.030 ${\pm}$ 0.036	0.544 ${\pm}$ 0.045	0.037 ${\pm}$ 0.019
Frozen twin-distance	$-$ 0.101 ${\pm}$ 0.054	0.238 ${\pm}$ 0.012	0.107 ${\pm}$ 0.075	0.366 ${\pm}$ 0.044	0.065 ${\pm}$ 0.051
BERTPE	0.073 ${\pm}$ 0.043	0.192 ${\pm}$ 0.010	0.175 ${\pm}$ 0.140	0.611 ${\pm}$ 0.052	0.044 ${\pm}$ 0.029
VAP3	0.092 ${\pm}$ 0.089	0.189 ${\pm}$ 0.034	0.186 ${\pm}$ 0.168	0.621 ${\pm}$ 0.095	0.059 ${\pm}$ 0.042
Embedding Specificity-IEF (PromptSET)	0.022 ${\pm}$ 0.005	6.102 ${\pm}$ 3.501	$-$ 0.011 ${\pm}$ 0.134	0.508 ${\pm}$ 0.002	0.030 ${\pm}$ 0.005

Appendix

Table 8: GlassGSM (real suite): UCD prediction on the audited, multi-operator-mix, pooled holdout, mean

{\pm}

sd over 5 seeds. Correlational score-predictor: predicts the base score

Y(J)

from text and uses it as a difficulty score; Embedding linear: RidgeCV on frozen embeddings regressed on

D_{\mathrm{avg}}

\rho_{S}

, NDCG@10, AUROC

\uparrow

; MAE,

|\rho_{C}|

shown for context.

Suite	Re-runs/item	Floor	Modal answer $\neq$ ref.
GlassGSM	4	$1.1\%$	$4$ ( $0.3\%$ )
GlassMATH	3	$2.5\%$	$0$ ( $0.0\%$ )

Appendix

Table 9: Identity-twin floor: break rate of re-solved, unperturbed base prompts on the solvable stratum (the average CCD of a no-effect perturbation).

Ranking	Recall@5%	Recall@10%	Recall@20%	Recall@50%
Random (expected)	0.05	0.10	0.20	0.50
Surface features (GBT)	0.066 ${\pm}$ 0.005	0.135 ${\pm}$ 0.012	0.251 ${\pm}$ 0.026	0.555 ${\pm}$ 0.044
Declared complexity	0.064 ${\pm}$ 0.007	0.122 ${\pm}$ 0.006	0.244 ${\pm}$ 0.007	0.567 ${\pm}$ 0.018
Frail- $\pi$ (ours)	0.099 ${\pm}$ 0.002	0.189 ${\pm}$ 0.005	0.359 ${\pm}$ 0.015	0.725 ${\pm}$ 0.020
$\widehat{D}^{\perp}$ (ours)	0.099 ${\pm}$ 0.002	0.186 ${\pm}$ 0.006	0.352 ${\pm}$ 0.021	0.676 ${\pm}$ 0.020
Oracle ( $\pi$ )	0.100 ${\pm}$ 0.001	0.195 ${\pm}$ 0.002	0.396 ${\pm}$ 0.004	0.981 ${\pm}$ 0.011

Appendix

Table 10: Recall of truly fragile jobs (

\pi>0.05

) in the top-

k\%

of the GlassGSM holdout ranked by each predictor, mean

{\pm}

sd over the same 5 seeds as Table 2 (373 held-out jobs/seed,

50.8\%

fragile on average). Random is the expected recall

k

; oracle ranks by the measured

\pi

itself. The practically meaningful quantity is the lift over random at small

k

	GlassMATH	GlassGSM
Pairs before audit	107,573	39,211
Kept (audited pool)	94,750	34,353
Dropped	12,823	4,858
gold-answer collision	12,059	4,738
entity collision	608	35
LLM contradiction	166	95

Appendix

Table 11: Contamination audit of the two real-math suites. “Kept” is the number of pairs entering the audited pool from which the benchmarks are drawn (both columns are the

m{=}9

generation files; the other dose files pass the same audit. The released corpora resample at most nine of these audited twins per problem at

m{=}9

and pool the five doses for Table 1 ); “before” is the raw un-audited count. The reason rows are not mutually exclusive — 11 GlassGSM pairs and 62 GlassMATH pairs are dropped for two reasons at once, so each reason row counts them again and the reasons sum to slightly more than Dropped .

Language Model Robustness

Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.

See all 506 papers

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations
Apr 2, 2026Shou-Tzu Han, Rodrigue Rizk, KC SantoshLanguage Model Robustness LLM Reasoning
Department of Computer Science, University of South Dakota
LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
May 14, 2026Philipp Mondorf, Samuel J. Bell, Jesse Dodge +1LLM Evaluation Neural Network Robustness
FAIR at Meta · LMU Munich · Munich Center for Machine Learning +1
Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
Jun 26, 2026Chenguang Wang, Ming Li, Xinyue Zeng +4Cognitive Modeling Educational Assessment
1Virginia Tech · University of Maryland · 3MBZUAI +1

Apr 2, 2026J/K move · Enter open · S save

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations

Shou-Tzu Han, Rodrigue Rizk, KC Santosh

Department of Computer Science, University of South Dakota

Language Model Robustness LLM Reasoning Adversarial Robustness of Language Models LLM Interpretability Causal Interventions in Language Models

Open paper AskPDF arXiv

Large language models demonstrate strong performance on mathematical reasoning benchmarks, yet remain surprisingly fragile to meaning-preserving surface perturbations. We systematically evaluate three open-weight LLMs, Mistral-7B, Llama-3-8B, and Qwen2.5-7B, on 677 GSM8K problems paired with semantically equivalent variants generated through name substitution and number format paraphrasing. All three models exhibit substantial answer-flip rates (28.8%-45.1%), with number paraphrasing consistently more disruptive than name swaps. To trace the mechanistic basis of these failures, we introduce the Mechanistic Perturbation Diagnostics (MPD) framework, combining logit lens analysis, activation patching, component ablation, and the Cascading Amplification Index (CAI) into a unified diagnostic pipeline. CAI, a novel metric quantifying layer-wise divergence amplification, outperforms first divergence layer as a failure predictor for two of three architectures (AUC up to 0.679). Logit lens reveals that flipped samples diverge from correct predictions at significantly earlier layers than stable samples. Activation patching reveals a stark architectural divide in failure localizability: Llama-3 failures are recoverable by patching at specific layers (43/60 samples), while Mistral and Qwen failures are broadly distributed (3/60 and 0/60). Based on these diagnostic signals, we propose a mechanistic failure taxonomy (localized, distributed, and entangled) and validate it through targeted repair experiments: steering vectors and layer fine-tuning recover 12.2% of localized failures (Llama-3) but only 7.2% of entangled (Qwen) and 5.2% of distributed (Mistral) failures.

Abstract

Language Model Robustness

Explore similar work

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations

LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction