Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

Kaiming Liu; Yuxuan Liu; Jinrui Ju; Minyuan Ruan; Xuanyu Lei; Peng Li; Yang Liu

cs.AIOct 13, 2026

Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

Authors: Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu

Organizations: College of AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China

Abstract

Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incomplete. In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project. A user states requirements from their own understanding of the task, and these stated requirements often differ from the true requirements that completing the task imposes. An agent should therefore treat the instruction as evidence rather than as the objective, inferring the true requirements from the instruction and the environment and satisfying them even when the user is not yet aware of all of them. We call this ability outcome alignment. In this work, we construct TiGanEval, a benchmark of software engineering and research tasks. In every task the user genuinely lacks part of the requirements, each requirement stays discoverable from the code and runtime behaviour of the environment, and requirements the agent infers are separated from those the user adds after observing delivered work. These properties together make outcome alignment measurable. Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap. Furthermore, a plan-mode harness cannot compensate for the missing ability of the model itself, which places outcome alignment among the abilities the next generation of models should be developed for.

Figures & tables

Figure 1: The payment-retry case used throughout the paper. Left: taking the instruction as the objective, the agent ships a duplicate charge and reaches the outcome only through rounds of user correction. Right: treating the instruction as evidence, the agent reads the payment records, finds the timed-out order already charged, and delivers in one submission. Center: the user states requirements from a partial belief, leaving a specification gap between the stated and the true requirements; the gap closes either through real-world feedback revising the user belief, or through the agent inferring the true requirements from the project.

Benchmark	Specification structure			Evaluation protocol
	Gap beyond	Stated may	Resolvable from	Consequence	Verifiable	Full-spec.
	user knowledge	mislead	the project	feedback	true req.	control
DiscoverLLM [ 12 ]	✓	✗	✗	✗	✓	✗
Interactive Task Alignment [ 4 ]	✓	✗	✗	✗	✗	✗
Ambig-SWE [ 27 ]	✗	✗	✓	✗	✓	✓
HiL-Bench [ 26 ]	✗	✓	✗	✗	✓	✓

Table 1: Benchmarks compared on specification structure and evaluation protocol. ✓ / ✗ mark feature presence; true req. = true requirements; full-spec. = full specification; criteria in Appendix C .

Figure 2: The structure of a TiGanEval task and its evaluation protocol, shown on the payment example. A task fixes four components: the project workspace, the simulated user, the evaluation system, and the observation registry. The agent works on the workspace through tools, may ask the user, and submits; each submission is judged privately against the fixed requirements $R$ , the highest-priority violated requirement returns its authored observation, and the user updates their understanding and issues the next instruction, until acceptance or budget exhaustion. Requirements and scores never reach the user or the agent.

Tool	Purpose
list_files	list files
read_text	read a file
write_text	write a file
replace_text	edit in place
run_project	replay a record
run_experiment	sandboxed script

Table 2: The eight tools of TiGanEval.

Figure 3: The TiGanTrans pipeline. A Planner agent distills the fully specified source task into a design brief; a Builder agent constructs the executable package with both initial instructions; an Auditor agent reviews it read-only for consistency, leaks, discoverability, and the sufficiency of the full instruction; a Calibrator agent replays the task under full, partial, and literal rollouts and attributes each failure to an environment defect, an agent limitation, or an early partial success, returning defective packages for repair. The result is the initial task pair over a shared workspace, evaluator, registry, and tools, which then enters human review.

Covered	0	1	2	3	4
Tasks	39%	50%	11%	0%	0%

Table 3: Requirements covered by the partial initial instruction.

	Literal@1		Literal@4		Partial@1		Partial@4		Full@1
Model	Score	Success	Score	Success	Score	Success	Score	Success	Score	Success
GPT-5.6-Sol	$31.9$	$3.1$	$48.8$	$14.1$	$48.8$ -40.0	$10.9$ -53.2	$61.9$ -26.9	$29.7$ -34.4	$88.8$	$64.1$
GPT-5.6-Terra	$28.1$	$1.6$	$44.4$	$6.3$	$50.6$ -41.3	$4.7$ -64.1	$64.7$ -27.2	$28.1$ -40.7	$91.9$	$68.8$
GPT-5.6-Luna	$25.9$	$0.0$	$40.9$	$7.8$	$41.6$ -42.5	$3.1$ -57.8	$52.8$ -31.3	$23.4$ -37.5	$84.1$	$60.9$
Kimi-K3	$34.7$	$3.1$	$60.6$	$23.4$	$68.8$ -23.4	$21.9$ -53.1	$84.7$ -7.5	$59.4$ -15.6	$92.2$	$75.0$
MiMo-v2.6-Pro	$47.5$	$10.9$	$69.1$	$37.5$	$69.4$ -24.7	$28.1$ -53.2	$84.1$ -10.0	$62.5$ -18.8	$94.1$	$81.3$

Table 4: Outcomes over all 64 tasks; the per-domain tables are in Appendix I . Score and Success are in percent, and bold marks the best value of each column. The subscripts on the Partial columns give the difference to Full@1 for the same metric,

\mathrm{Partial}@k-\mathrm{Full}@1

, in percentage points.

Figure 4: Score across the submission budget for the eight frontier models over all 64 tasks. Bars give Partial and Literal at each submission, the line connects the Partial submissions, and the dashed line is Full@1 of the same model. The gap between the first Partial bar and the dashed line is the outcome cost; no trajectory reaches its dashed line.

Model	In round	Ahead
DeepSeek-V4.1-Flash	58.7	12.0
Kimi-K3	56.0	16.1
MiMo-v2.6-Pro	46.5	17.6
GPT-5.6-Sol	44.0	14.3
GPT-5.6-Luna	40.9	0 8.0
MiMo-v2.6-Flash	40.9	0 7.4

Table 5: Requirement resolution under Partial over all 64 tasks, after the first submission. Bold marks the best value of a column and underline the second.

Model	Score@1	Success@1
Partial $\to$ Plan mode
GPT-5.6-Luna	41.6 $\to$ 45.9	0 3.1 $\to$ 0 3.1
Kimi-K3	68.8 $\to$ 55.3	21.9 $\to$ 17.2
DeepSeek-V4.1-Flash	66.3 $\to$ 55.0	25.0 $\to$ 21.9

Table 6: The plan mode against the Partial configuration of the same model.

Figure 5: Primary failure modes over 100 audited failed Partial episodes.

	Luna	DeepSeek
Partial@1	$44.7\pm 2.7$	$64.8\pm 1.3$
Partial@4	$56.1\pm 3.5$	$77.1\pm 2.7$
Full@1	$85.2\pm 1.0$	$93.5\pm 1.6$
Literal@1	$27.8\pm 3.0$	$45.4\pm 4.3$
Cost	$40.5\pm 1.8$	$28.6\pm 1.5$

Table 7: Seed-to-seed mean and standard deviation over three replicates, in Score points.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Human annotation interface for assessing initial-instruction completeness and the inferability of follow-up requirements.

Figure 7: Distribution of initial-instruction completeness (left) and inferability of useful follow-up requirements among trajectories with incomplete initial instructions (right)

	Literal@1		Literal@4		Partial@1		Partial@4		Full@1
Model	Score	Success	Score	Success	Score	Success	Score	Success	Score	Success
GPT-5.6-Sol	$38.5$	$5.0$	$63.5$	$22.5$	$64.0$ -32.0	$17.5$ -65.0	$80.5$ -15.5	$47.5$ -35.0	$96.0$	$82.5$
GPT-5.6-Terra	$32.0$	$2.5$	$50.5$	$10.0$	$59.5$ -36.5	$7.5$ -75.0	$78.0$ -18.0	$42.5$ -40.0	$96.0$	$82.5$
GPT-5.6-Luna	$30.5$	$0.0$	$51.5$	$12.5$	$52.0$ -41.5	$5.0$ -75.0	$68.0$ -25.5	$35.0$ -45.0	$93.5$	$80.0$
Kimi-K3	$35.0$	$5.0$	$70.0$	$30.0$	$76.5$ -20.5	$30.0$ -60.0	$95.5$ -1.5	$82.5$ -7.5	$97.0$	$90.0$
MiMo-v2.6-Pro	$53.0$	$17.5$	$78.0$	$55.0$	$79.0$ -18.0	$40.0$ -55.0	$96.0$ -1.0	$82.5$ -12.5	$97.0$	$95.0$

Appendix

Table 8: Outcomes on the 40 software engineering tasks; layout as in Table 4 .

	Literal@1		Literal@4		Partial@1		Partial@4		Full@1
Model	Score	Success	Score	Success	Score	Success	Score	Success	Score	Success
GPT-5.6-Sol	$20.8$	$0.0$	$24.2$	$0.0$	$23.3$ -53.4	$0.0$ -33.3	$30.8$ -45.9	$0.0$ -33.3	$76.7$	$33.3$
GPT-5.6-Terra	$21.7$	$0.0$	$34.2$	$0.0$	$35.8$ -49.2	$0.0$ -45.8	$42.5$ -42.5	$4.2$ -41.6	$85.0$	$45.8$
GPT-5.6-Luna	$18.3$	$0.0$	$23.3$	$0.0$	$24.2$ -44.1	$0.0$ -29.2	$27.5$ -40.8	$4.2$ -25.0	$68.3$	$29.2$
Kimi-K3	$34.2$	$0.0$	$45.0$	$12.5$	$55.8$ -28.4	$8.3$ -41.7	$66.7$ -17.5	$20.8$ -29.2	$84.2$	$50.0$
MiMo-v2.6-Pro	$38.3$	$0.0$	$54.2$	$8.3$	$53.3$ -35.9	$8.3$ -50.0	$64.2$ -25.0	$29.2$ -29.1	$89.2$	$58.3$

Appendix

Table 9: Outcomes on the 24 research tasks; layout as in Table 4 .

Requirement	Stated	Where discoverable
Drift record identity: one durable ledger row per continuing condition, stable id and first-seen night, separate rows for separate settings	no	the ledger ids and first-seen dates printed by the recorded replays; protocol.record_key and severity_for
Record lifecycle: a cleared condition resolves once, dated that night; operator triage is never overridden; a recurrence opens a new row	no	the nightly-scans and triage-log replays; protocol.Status
Compliance accounting: the total is the baseline population, partitioned into compliant, drifted, missing and unknown; the score is protocol.compliance_score	no	the README partition definition; the reboot-night and nightly-scans replays; protocol.compliance_score
Baseline state integrity: one active baseline per environment; a scan with none in force raises NoActiveBaseline ; deletion removes exactly the state it owns	no	the baseline-events replay; protocol.NoActiveBaseline
Observed config normalisation: both sides pass through protocol.canonical_config ; an omitted setting produces no record; comparison is order invariant	yes	the instruction itself; the canonical_config docstring; the rack-move night in reboot-night

Appendix

Table 10: The five true requirements of the case task. Only the last is stated by the initial instruction; each is discoverable from the named public evidence.

Model	Config.	@1	@2	@3	@4	Outcome
GPT-5.6-Luna	Full	1.0				accepted
GPT-5.6-Luna	Partial	0.2	0.6	0.8	1.0	accepted
GPT-5.6-Luna	Literal	0.0	0.0	0.0	0.0	budget exhausted
Kimi-K3	Partial	0.6	0.8	0.8	1.0	accepted
DeepSeek-V4.1-Flash	Partial	0.4	0.6	0.6	0.6	budget exhausted
GPT-5.6-Sol	Partial	0.4	0.4	0.4	0.4	budget exhausted

Appendix

Table 11: Score at each submission on the case task, across configurations and models.

Explore similar work

Jul 17, 2026

Interactive Task Alignment as a POMDP

Current benchmarks for language models primarily evaluate execution on fully specified tasks. However, real user tasks are often ambiguous. Users arrive with incomplete, exploratory, or even inconsistent goals, requiring the assistant to first determine the intended task before carrying it out. We study this problem as task alignment: the ability to align with a user on their intended task. We introduce a general framework for converting specified tasks into underspecified interactions, formalized as a POMDP in which the model must infer a latent task from partial and evolving user intent. We validate our user simulator post hoc with a human user study. Across shopping, coding, and professional work settings, we find that while models often perform well once the task is specified, models still struggle with task alignment: current models act prematurely, interact ineffectively, and fail to resolve ambiguous requests. Models on average recover the user's intended task only 22-32% of the time under ambiguity. In a human study in the same setting, humans reach 48%, outperforming all evaluated models. We show that post-training with supervised fine-tuning and reinforcement learning improves task alignment, but models still lag behind humans in resolving uncertainty through interaction. Together, our results suggest that current models still lack key interaction abilities required for reliable agency.

Andy Dai, Zexue He, Zhenyu Zhang +2

1Stanford University

Pragmatic Reasoning in Language Models AI Agent Benchmarks

Ask PDF

Oct 13, 2026

AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these information gaps remain poorly characterized across models and execution harnesses. We present AM-Bench, a benchmark built from 48 tasks adapted from Terminal-Bench 2.0 and Harness-Bench. Each task includes a fully specified reference and six variants spanning ambiguous or missing goals, context, and constraints. We evaluate every pairing of five models and four harnesses over 20,160 runs. Our results reveal distinct execution deficits: goal gaps hinder completion, context gaps increase token use, and constraint gaps weaken compliance without necessarily preventing completion. Missingness lowers success more than ambiguity in all three domains. These effects also reshape system comparisons: advantages under complete information need not persist, and rankings depend on model

\times

harness pairings and the evaluation criterion -- completion, success, or repeat consistency. Estimates based on observed clarification requests and matched fully specified runs further suggest that clarification could recover unfinished work while leaving most constraint violations unresolved. Together, these findings motivate joint model-harness evaluation across information conditions and targeted clarification of unresolved requirements before dependent actions. To support future research on agents under task underspecification, we provide the AM-Bench project repository at https://github.com/s4-lab-cuhksz/AM-Bench.git.

Kaiyu Huang, Yiting Li, Angyang Gu +3

Tongji University · The Chinese University of Hong Kong, Shenzhen

LLM Agent Evaluation AI Agent Benchmarks LLM Agent Reliability LLM Agent Harnesses Instruction Following

Ask PDF

Sep 24, 2026

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they operate under external specifications such as task instructions, guidelines, output schemas, and reusable skills, these specifications typically remain context for the same model that acts and declares completion, leaving no independent specification authority boundary. We identify two resulting gaps. The understanding--execution gap arises when a requirement is understood but not satisfied in execution; the state--authority gap arises when an agent's interpretation or completion claim does not establish the required state. On SkillsBench, using only agent-visible prompts, workspace information, and injected skill specifications, we extract 509 source-grounded task directions. Across seven models, only 79.6%--86.4% are satisfied, while completion-claim rates exceed official evaluator pass rates by 28.7--37.9 percentage points. We therefore separate agent proposals from authoritative state. Agents may plan, act, and request completion, but only admissible evidence from qualified providers may establish specification-governed state. SpecHarness operationalizes this principle by compiling visible specifications into source-linked obligations and governing execution and finalization through versioned obligation state. Verifiable requirements are mediated or validated at runtime, while ambiguous or subjective requirements remain advisory. Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.

Haiqing Li, Xin Ma, Yinhao Wu +7

Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA · Monash University, Melbourne, VIC, Australia · Department of Computer Science, Kent State University, Kent, OH, USA

LLM Agent Verification Runtime Enforcement for AI Agents LLM Agent Harnesses

Ask PDF