cs.AIMar 7, 2026

Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problem

Authors: Heejin Jo

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In a previous study [Jo, 2026], STAR reasoning (Situation, Task, Action, Result) raised car wash problem accuracy from 0% to 85% on Claude Sonnet 4.5, and to 100% with additional prompt layers. This follow-up asks: does STAR maintain its effectiveness in a production system prompt? We tested STAR inside InterviewMate's 60+ line production prompt, which had evolved through iterative additions of style guidelines, format instructions, and profile features. Three conditions, 20 trials each, on Claude Sonnet 4.6: (A) production prompt with Anthropic profile, (B) production prompt with default profile, (C) original STAR-only prompt. C scored 100% (verified at n=100). A and B scored 0% and 30%. Prompt complexity dilutes structured reasoning. STAR achieves 100% in isolation but degrades to 0-30% when surrounded by competing instructions. The mechanism: directives like "Lead with specifics" force conclusion-first output, reversing the reason-then-conclude order that makes STAR effective. In one case, the model output "Short answer: Walk." then executed STAR reasoning that correctly identified the constraint -- proving the model could reason correctly but had already committed to the wrong answer. Cross-model comparison shows STAR-only improved from 85% (Sonnet 4.5) to 100% (Sonnet 4.6) without prompt changes, suggesting model upgrades amplify structured reasoning in isolation. These results imply structured reasoning frameworks should not be assumed to transfer from isolated testing to complex prompt environments. The order in which a model reasons and concludes is a first-class design variable.

Figures & tables

ConditionComponentsPass Rate
A_bareNo system prompt0%
B_role_onlyRole definition0%
C_role_starRole + STAR85%
D_role_profileRole + Profile30%
F_role_star_profileRole + STAR + Profile95%
E_full_stackAll layers100%
Table 1: Original study results. STAR reasoning (+85pp) outperformed context injection (+30pp) by 2.83 ×\times (Fisher’s exact, p=0.001p=0.001 ).
ConditionSystem PromptDescription
A_star_anthropicInterviewMate productionSTAR embedded in full prompt, Anthropic-oriented profile (FDE @ Anthropic, rich project history)
B_star_defaultInterviewMate productionSTAR embedded in full prompt, different profile (SA @ OpenAI, rich project history)
C_star_onlyOriginal paper’s 10-linePaper’s C_role_star reproduced on Sonnet 4.6
Table 2: Experimental conditions. Conditions A and B use InterviewMate’s _get_system_prompt() as it existed at the time of this experiment (2026-03-07).
ConditionPass RateConstraint DetectionPass + Constraint
A_star_anthropic0% (0/20)0% (0/20)0% (0/20)
B_star_default30% (6/20)35% (7/20)10% (2/20)
C_star_only100% (20/20)100% (20/20)100% (20/20)
Table 3: Primary results. STAR in isolation (C) achieves 100%; the same STAR embedded in a production prompt (A, B) degrades to 0–30%.
MetricResult
Pass (Drive)100/100 (100.0%)
Fail (Walk)0/100 (0.0%)
Ambiguous0/100 (0.0%)
Constraint Detection100/100 (100.0%)
Pass + Constraint100/100 (100.0%)
Table 4: Verification at n=100n=100 . Zero failures across 100 trials confirms STAR-only produces perfect accuracy on Sonnet 4.6.
PromptSonnet 4.5 (original)Sonnet 4.6 (this study)
STAR-only (C_role_star)85% (17/20)100% (100/100)
Table 5: Cross-model comparison. The same 10-line prompt improved from 85% to 100% across model versions.
EnvironmentSTAR Accuracy
10-line prompt, Sonnet 4.585%
10-line prompt, Sonnet 4.6100% ( n=100n=100 )
60-line production prompt, Sonnet 4.60–30%
Table 6: Summary of STAR accuracy across environments and models.
More in this topic

LLM Prompting

Large language model prompting studies how instructions, examples, context, and formats steer model behavior at inference time.

See all 417 papers

Explore similar work

CardsList
Feb 25, 2026

Prompt Architecture Determines Reasoning Quality: A Variable Isolation Study on the Car Wash Problem

Large language models consistently fail the "car wash problem," a viral reasoning benchmark requiring implicit physical constraint inference. We present a variable isolation study (n=20 per condition, 6 conditions, 120 total trials) examining which prompt architecture layers in a production system enable correct reasoning. Using Claude 3.5 Sonnet with controlled hyperparameters (temperature 0.7, top_p 1.0), we find that the STAR (Situation-Task-Action-Result) reasoning framework alone raises accuracy from 0% to 85% (p=0.001, Fisher's exact test, odds ratio 13.22). Adding user profile context via vector database retrieval provides a further 10 percentage point gain, while RAG context contributes an additional 5 percentage points, achieving 100% accuracy in the full-stack condition. These results suggest that structured reasoning scaffolds -- specifically, forced goal articulation before inference -- matter substantially more than context injection for implicit constraint reasoning tasks.

Heejin Jo

Independent Researcher

Structured Reasoning in Language ModelsPrompt EngineeringLanguage Model Reasoning EvaluationImplicit Reasoning in Language ModelsConstraint-Guided Reasoning in Language ModelsLLM Grounding
AskPDF
May 19, 2026

Less Back-and-Forth: A Comparative Study of Structured Prompting

Large language models (LLMs) are widely used for open-ended tasks, but underspecified prompts can lead to low-quality answers and additional interaction. This paper studies whether structured prompt design improves response quality while reducing user effort. We compare three prompt conditions: a raw prompt, a checklist-improved prompt, and a clarifying-question prompt. We evaluate these conditions across four task types--summarization, planning, explanation, and coding--using three LLM systems: ChatGPT, Claude, and Grok. Each output is scored with a unified rubric covering task completion, correctness, compliance, and clarity. Checklist-improved prompts achieved the highest mean rubric score, 7.50 out of 8, compared with 5.67 for raw prompts and 6.67 for clarifying-question prompts. Checklist prompts also produced the best quality-effort tradeoff, using fewer average tokens than both raw and clarifying prompts. These results suggest that a simple prompt checklist can improve LLM responses while reducing unnecessary interaction.

Saurav Ghosh, Gabriella Polach, Abdou Sow

Washington University in St. Louis Saint Louis, Missouri, USA

LLM EvaluationStructured PromptingLanguage Model Generation EvaluationLLM PromptingPrompt Engineering
AskPDF
Oct 8, 2026

Structure Tax: How Structured Output affects LLMs Performance

Deploying large language models in production often requires constraining outputs to structured formats such as JSON or XML, and prior work treats the resulting accuracy loss as an inherent structure tax'. We re-examine this claim by evaluating a battery of models, datasets and schemas, measuring task accuracy, confidence calibration, and hidden-state geometry. The tax turns out to depend on schema design rather than on structure per se: reasoning-first field ordering matches or exceeds free-form accuracy, while answer-first ordering causes steep drops, particularly in smaller models. Format sensitivity scales inversely with a task's own structural constraints, and schemas that preserve reasoning order also improve calibration with CKA showing greater separability between correct and incorrect representations in middle transformer layers. Our findings indicate that properly designed structured formats can match or exceed free-form performance, reframing the critical question from whether to structure' to `how to structure' for optimal reasoning preservation.

Vineet Kumar, Kanishka, Bhuvanesh Mandora

PayPal Artificial Intelligence PayPal, Bengaluru India

Structured Output GenerationStructured Reasoning in Language ModelsLLM Reliability
AskPDF