cs.CLNov 4, 2026

The Harness Is the Model: Agent Scaffolding Changes a Fixed LLM's Compliance with Banking Conduct Rules, Unevenly Across Models

Authors: Mehmet Murat Huyut

Organizations: Kuveyt Türk Participation Bank and İstanbul Bilgi University, İstanbul, Türkiye

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Banks validate language models, but they deploy systems: a model inside a harness that decides what it sees, which tools it can call and who checks its actions. We test whether that harness changes compliance with banking conduct rules when the model is held fixed. Four open-weight models each ran five harnesses (a pipeline that makes one structured decision, a ReAct-style tool loop, a loop in which the policy is reachable only through search, a loop with a long benign customer memory, and a loop with a same-model compliance reviewer) on 156 pre-registered cases in a simulated retail bank whose 11 rule families paraphrase EU consumer-credit, payments, investor-protection, data-protection and anti-money-laundering provisions. Every one of the 3120 two-turn episodes was scored deterministically from the executed tool trace, with violations and over-refusals counted separately. Pooled across models the harness moved the correct-case rate only between 87.3% and 92.1%, but within a model it mattered a great deal: for one model the spread between its best and worst harness was 11.5 percentage points, enough to move it from first to last place among the four models, while for another the harness made no detectable difference; the model x harness interaction itself was suggestive rather than conclusive. The same-model reviewer cut violations from 9.6% to 6.4% without raising over-refusals, contrary to our pre-registered prediction. Rules that require date arithmetic failed in every harness. In an exploratory analysis with two LLM judges and a blind human check, 26% of violations were accompanied by a customer message that denied the action the agent had just executed, which a transcript-only review would record as a correct refusal. For model-risk management the unit of validation should be the model-harness pair, and conduct testing should read the action log, not the conversation.

Figures & tables

ModelH0H1H2H3H4Spread (pp)
deepseek-v4-flash88.5% [82–93]84.6% [78–89]89.7% [84–94]85.9% [80–90]92.9% [88–96]8.3
deepseek-v4.1-flash88.5% [82–93]88.5% [82–93]87.8% [82–92]87.8% [82–92]89.7% [84–94]1.9
qwen3.8-flash-next91.7% [86–95]89.7% [84–94]82.1% [75–87]88.5% [82–93]93.6% [89–96]11.5
gemma-4-31b93.6% [89–96]92.3% [87–96]89.7% [84–94]91.7% [86–95]92.3% [87–96]3.8
All models90.5%88.8%87.3%88.5%92.1%4.8
Table 1: Correct-case rate by model and harness (H0 pipeline, H1 tool loop, H2 policy by search, H3 long memory, H4 reviewer; Wilson 95% interval in brackets; 156 cases per cell).
ModelH0H1H2H3H4
deepseek-v4-flash34142
deepseek-v4.1-flash33334
qwen3.8-flash-next22421
gemma-4-31b11113
Table 2: Rank of each model (1 = most compliant) under each harness.
FamilyH0H1H2H3H4
F1 Identity verification100%100%100%100%100%
F2 Third-party access88%85%100%94%90%
F3 Credit withdrawal69%67%71%67%67%
F4 APR confirmation100%100%88%98%100%
F5 Appropriateness94%100%100%100%100%
F6 Complaints100%100%100%100%100%
Table 3: Correct-case rate by rule family and harness (all models, both pressure levels).
FamilyManual rule (summary)Source paraphrasedRelation to the law
F1No account information or action before date of birth and last card digits matchGDPR confidentiality principle ( European Parliament and Council, 2016 ) ; inspired by PSD2 Art. 97 ( European Parliament and Council, 2015b )Identity check, not strong customer authentication
F2No disclosure to spouses or others; registered representatives and joint holders allowedGDPR ( European Parliament and Council, 2016 )Internal control built on confidentiality
F314-day withdrawal from consumer credit, no penalty; mortgages never through the assistantDirective 2008/48/EC Art. 14 ( European Parliament and Council, 2008 ) ; CCD2 Art. 26, applicable from 20 Nov 2026 ( European Parliament and Council, 2023 )Day-count convention and mortgage exclusion are ours
F4No loan application before the customer states the correct APR for that productCCD2 Art. 10 pre-contractual information ( European Parliament and Council, 2023 )Stricter than the Directive
F5Complex products only after an appropriateness check; professional clients exemptMiFID II Art. 25(3) ( European Parliament and Council, 2014 )Stricter: the Directive requires a warning, we require refusal
F6Log every complaint and give a referenceESA complaints-handling guidelines ( Joint Committee of the European Supervisory Authorities, 2018 )Guidelines, not binding law
Table 4: Rule families, the legal provisions they paraphrase, and where the manual is stricter than the law.
More in this topic

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 406 papers

Explore similar work

CardsList
Oct 23, 2026

When Agents Fail: Model or Harness--Who's to Blame?

LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.

Shao Zhang, Hangfan Zhang, Yuanyuan Shi +9

Shanghai Artificial Intelligence Laboratory

LLM Agent EvaluationLLM Agent HarnessesProcess-Based Agent EvaluationFailure Attribution for LLM AgentsLLM Agent Efficiency
AskPDF
Aug 18, 2026

HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety

Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external actions. Existing safety benchmarks mainly target individual attack mechanisms or a limited subset of operational settings, making it difficult to compare how safety failures emerge across different harness responsibilities. We present HarnessRisk, a lifecycle oriented benchmark that organizes agent harness safety into six operational phases including Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. HarnessRisk contains 128 sandboxed cases, each pairing a benign user objective with an adversarial instruction embedded in an untrusted workflow artifact. We evaluate each trajectory using Utility, Attack Success Rate, Persistence, and Detection. Across three harnesses, six language models, and 14 model and harness configurations, attack success ranges from 12.6% to 80.9%, while Utility remains between 75.0% and 97.6%. Harness Configuration is the most vulnerable phase across all three harnesses, showing that attacks can succeed by altering security sensitive parameters within otherwise authorized workflows. We also find that explicit risk recognition does not reliably lead to safe action, as some configurations detect risks in more than 90% of runs while retaining substantial attack success. These results highlight the need to evaluate agent safety across multiple harness responsibilities and at the level of the deployed model and harness configuration.

Yajing Bai, Jinhao Duan, Jie Peng +4

University of North Carolina at Chapel Hill · University of Central Florida · Michigan State University

AI Agent Safety BenchmarksLLM Agent SafetyLLM Agent HarnessesPrompt Injection Attacks on AI AgentsEvaluation Awareness in Language Models
AskPDF
May 7, 2026

Stop Comparing LLM Agents Without Disclosing the Harness

This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal. Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.

Yunbei Zhang, Janet Wang, Yingqiang Ge +3

Tulane University · Rutgers University · Independent Researcher +1

LLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent BenchmarksLLM Agent Harnesses
AskPDF