A Regression Framework for Understanding Prompt Component Impact on LLM Performance

Andrew Lauziere; Jonathan Daugherty; Taisa Kushner

cs.LGMar 27, 2026

A Regression Framework for Understanding Prompt Component Impact on LLM Performance

Authors: Andrew Lauziere, Jonathan Daugherty, Taisa Kushner

Organizations: Galois, Inc.

Abstract

As large language models (LLMs) continue to improve and see further integration into software systems, so does the need to understand the conditions in which they will perform. We contribute a statistical framework for understanding the impact of specific prompt features on LLM performance. The approach extends previous explainable artificial intelligence (XAI) methods specifically to inspect LLMs by fitting regression models relating portions of the prompt to LLM evaluation. We apply our method to compare how two open-source models, Mistral-7B and GPT-OSS-20B, leverage the prompt to perform a simple arithmetic problem. Regression models of individual prompt portions explain 72% and 77% of variation in model performances, respectively. We find misinformation in the form of incorrect example query-answer pairs impedes both models from solving the arithmetic query, though positive examples do not find significant variability in the impact of positive and negative instructions - these prompts have contradictory effects on model performance. The framework serves as a tool for decision makers in critical scenarios to gain granular insight into how the prompt influences an LLM to solve a task.

Figures & tables

Fig. 1 : Overview of IAMs. First in panel A, a prompt model describes texts of interest which could impact LLM performance on the query “3+2=” (blue). The first stratum (purple) contains two choices of task-specific description texts while five strata (red) each have an example query-answer pair. An LLM processes and scores the full set of possible subprompts, each an input prompt containing a subset of prompt components (panel B). The inclusion (modeled with a 1) or exclusion (modeled with a 0) of components is expressed with binary variables (panel C). Pairs of encoding vectors and scores are used to fit a regression model, such as the one written in panel D. Panel E shows an example visualization of the learned model parameters conveying importance for LLM performance.

Panel of Figure 2 — Fig. 2 : GPT-OSS-20B was more sensitive to the prompt than Mistral-7B. Left: Mistral-7B’s DCPMI across all 8192 subprompts (red) showed a unimodal distribution whereas GPT-OSS-20B’s DCPMI distribution was bimodal (black). The baseline DCPMI, 1.0 for both models, stands at the center of the Mistral-7B distribution. Right: Quantile-quantile (QQ) plots for Mistral (residuals in red, line of best fit in blue) and OSS (residuals in black, line of best fit in green) from first-order regression models. Overall, the unary model for OSS yielded more extreme residuals (y-axis values below -3 and above 3) than Mistral. The OSS residual distribution also showed deviation from normality as the tails of the distribution had less mass than expected (black dots under the green line).

	Mistral [ 15 ]	OSS [ 25 ]
Component	Coefficient	Coefficient
Intercept	1.604 ***	3.324 ***
_	-0.020	-0.231 ***
Pretend you’re a …	0.045 ***	-3.057 ***
To do addition, …	0.176 ***	0.130 ***
To do subtraction, …	-0.037 **	-0.774 ***

TABLE I : Mistral-7B and GPT-OSS-20B similarly used example query-answer pairs to solve the query, but instruction texts derailed OSS. Prompt components ( Component ) and corresponding first-order parameter estimates ( Coefficient ) for both Mistral and OSS explain how each model solved the query. Positive or true components are marked in light-green background, negative or false components are marked with red background, and the neutral underscore is marked in gray. ***: p < 0.0001, **: p < 0.005, *: p < 0.05.

Fig. 3 : LASSO trades off fidelity for interpretability. LASSO penalized model size (green curves, measured $\|\hat{\boldsymbol{\beta}}_{\lambda}\|_{2}$ ) with increasing $\lambda$ . As models shrank, performance lowered and MSE (blue curves) rose, yielding a trade-off. The grid search procedure identified an optimal $\lambda^{*}\approx 0.0004$ for both models. At these “elbows” of the green curves, model reduction began to slow.

Fig. 4 : LASSO penalized interaction terms more than first-order terms in both Mistral and OSS. Model sizes, $\|\hat{\boldsymbol{\beta}}_{\lambda}\|_{2}$ , were stratified by level of interaction. Blue curves show Intercept (baseline) estimates for each $\lambda$ ; OSS first-order estimates remained larger than the baseline whereas all prompt component parameter estimates fell below the baseline value for Mistral, suggesting that OSS used prompt components more than Mistral, relative to baseline query DCPMIs.

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 429 papers

PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs
Jun 21, 2026Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari +2LLM Evaluation Language Model Generation Evaluation
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan
How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs
Sep 28, 2026Ali Alfageeh, Rahul Gopinath, Amin AlipourLLM Prompting Few-Shot Prompting
University of Houston · The University of Sydney
Shared Lexical Task Representations Explain Behavioral Variability In LLMs
Apr 23, 2026Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez +5Prompt Sensitivity LLM Prompting
Brown University · MIT · Northeastern

Jun 21, 2026J/K move · Enter open · S save

PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs

Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari, Gibrail Islam+1

School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan

LLM Evaluation Language Model Generation Evaluation LLM Prompting LLM Reliability

Open paper AskPDF arXiv

Large language models (LLMs) often encounter conflicting prompts, although current instruction following benchmarks assess those meta-instructions in isolation, limiting the insights about how models process conflicting instructions. We introduce a framework PRIME(Prompt Resolution under Incompatible Meta-Instructions Evaluation) to analyze behavior of LLMs when provided with conflicting instructions. PRIME purposefully produces calibrated conflicts across response length, output format, and reasoning; classifying model responses with a deterministic behavioral taxonomy. We are evaluating five instruction tuned open weight LLMs in two distinct settings, balanced and naturally distributed. The conclusion we reach upon analysis is that conflict type is more significant in affecting behavior than model scale, and various failure modes across different categories of conflict. Our findings emphasize the value of developing conflict awareness and suggest ability of LLM to follow instructions cannot be assessed through isolated constraints alone.

Abstract

LLM Prompting

Explore similar work

PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs

How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs

Shared Lexical Task Representations Explain Behavioral Variability In LLMs