Software World Models: From Consequence Prediction to Decision Value

Tongli Su; Yuntong Hu; Liang Zhao; Bowen Zhu; JayaSai Somasundaram; Hasibul Haque

cs.LGOct 4, 2026

Software World Models: From Consequence Prediction to Decision Value

Authors: Tongli Su, Yuntong Hu, Liang Zhao, Bowen Zhu, JayaSai Somasundaram, Hasibul Haque

Organizations: Emory University

Abstract

A coding agent may safely modify one repository while silently breaking downstream services, libraries, or datastores that depend on it. Exhaustively running integration tests after every agent action is impractical, so the agent must predict these failures before executing them. Existing software world models predict the agent's own observations, while static change-impact analysis only identifies where a change may propagate. We instead introduce the Software World Model (SWM), which models the broader system affected by a code change and predicts its blast set: the components that the change will break. SWM follows three stages: explore, learn, and act. Explore executes candidate changes from restored system states, prioritizing regions where observed failures contradict the dependency graph. Learn fine-tunes a language model on these execution outcomes to predict downstream breakage. Act converts sampled predictions into per-consumer break probabilities for change ranking, proactive migration, and deciding when another execution is worth its cost. On held-out synthetic systems, SWM improves blast-set F1 from 0.431 for static reachability to $0.571\pm0.037$ , more than halves ranking regret, and improves migration return at all nine evaluation checkpoints. The two methods are complementary: reachability is stronger on dependencies represented in the graph, while SWM recovers failures caused by couplings the graph misses. Experiments on held-out real libraries further show that predicting structured failure outcomes, rather than only scalar risk, is important for downstream decision quality.

Figures & tables

Figure 1: SWM closes the loop from exploring executable software worlds, to learning their change-conditioned consequences, to improving the coding agent that acts in them.

Figure 2: Companion evidence across Explore, Learn, and Act. (a) Fractions of locally passing real-library perturbations that break a consumer, contradict static prediction, or expose a hidden channel. (b) Blast-set F1 on 90 held-out synthetic forecasts; dots mark the mean and mean

\pm

three-seed half-range. (c) Explicit-belief planning on 40 migration tasks, with perfect foresight as a reference. The panels use separate evaluation populations.

Figure 3: Exploration, consequence learning, and agent improvement. The prediction study in (b) reports three-seed half-ranges; the agent study in (c) reports paired world-bootstrap 95% intervals and individual seed means.

Agent	Synthetic			Real libraries
	Return $\uparrow$	Recall $\uparrow$	Broken $\downarrow$	Return $\uparrow$	Recall $\uparrow$	Broken $\downarrow$
Base agent	$0.721$	$0.444$	$0.600$	$0.453$	$0.276$	$0.387$
CWM	${0.740}_{+0.019}$	$0.533$	$0.499$	${0.548}_{+0.095}$	$0.381$	$0.351$
Qwen- AgentWorld	${0.754}_{+0.033}$	$0.559$	$0.447$	${0.571}_{+0.118}$	$0.446$	$0.360$
SWM (Ours)	${0.773}_{+0.052}$	$0.586$	$0.363$	${0.613}_{+0.160}$	$0.499$	$0.309$

Table 1: Mean agent performance on synthetic and real-library worlds. Return subscripts give gains over the base agent. Standard deviations appear in Table 5 .

Training corpus	Return $\uparrow$	$\Delta$ return [95% interval]
Base agent	0.721	n/a
Random	0.772	$+0.050\;[+0.021,+0.088]$
Coverage	0.768	$+0.046\;[+0.012,+0.089]$
Expert	0.780	$+0.059\;[+0.029,+0.095]$
Decision-aware	0.765	$+0.044\;[+0.010,+0.087]$

Table 2: Consequence training improves the base agent on 40 matched tasks. Trained rows average three seeds at 360 records; brackets give paired world-bootstrap 95% intervals for return differences, computed before rounding.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Evaluation	Score	Comparator or context	Gain or evidence
Explore	Real hidden-channel yield	$0.244$	random: $0.157$	$+55\%$
Learn	Synthetic blast-set F1 (supervised recipe)	$0.608\pm 0.016$	reachability: $0.431$	$+0.177$
Learn	Synthetic hidden-channel F1	$0.304$	reachability: $0.189$	$+61\%$
Learn	Real-library hidden-channel F1	$0.198\pm 0.038$	reachability: $0.137$	$+45\%$
Act	Candidate-ranking regret $\downarrow$	$0.593$	reachability: $1.430$	$-59\%$
Act	Belief-plan return $\uparrow$	$0.757$	reachability: $0.685$	$+0.072$

Appendix

Table 3: Exploration, forecast, and explicit-decision results. Each row uses its stated population and comparator.

Figure 4: Exploration affects prediction and ranking differently. (a) Blast-set F1 and (b) ranking regret across execution budgets; points are three-seed means and error bars are half-ranges. (c) The 100-record comparison. Coverage and expert exploration were not evaluated at 360 records; the first 60 decision-aware records are random. Higher F1 and lower regret are better.

Corpus	Records	Seed	Return	$\Delta$ return	95% interval
Random	100	0	0.749	$+0.028$	$[-0.009,+0.072]$
Random	100	1	0.754	$+0.033$	$[+0.005,+0.067]$
Random	100	2	0.731	$+0.010$	$[-0.023,+0.052]$
Random	200	0	0.755	$+0.034$	$[+0.011,+0.065]$
Random	200	1	0.769	$+0.048$	$[+0.018,+0.086]$
Random	200	2	0.778	$+0.056$	$[+0.025,+0.094]$

Appendix

Table 4: All 24 trained arms on the same 40 tasks; base-agent return is

0.721

. Intervals are the source campaign’s per-seed paired world-cluster 95% intervals.

Figure 5: All twelve full-budget training seeds improve agent return, candidate recall, and services left broken relative to the base checkpoint. Panel (a) includes paired world-cluster 95% intervals; (b–c) show per-seed mean changes on the same tasks.

Agent	Synthetic worlds			Real libraries
	Return $\uparrow$	Recall $\uparrow$	Broken $\downarrow$	Return $\uparrow$	Recall $\uparrow$	Broken $\downarrow$
Base agent	$0.721$	$0.444$	$0.600$	$0.453$	$0.276$	$0.387$
CWM ( Copet et al., 2025 )	${0.740}_{+0.019}^{\pm 0.006}$	$0.533^{\pm 0.007}$	$0.499^{\pm 0.020}$	${0.548}_{+0.095}^{\pm 0.005}$	$0.381^{\pm 0.005}$	$0.351^{\pm 0.014}$
Qwen-AgentWorld ( Zuo et al., 2026 )	${0.754}_{+0.033}^{\pm 0.006}$	$0.559^{\pm 0.022}$	$0.447^{\pm 0.027}$	${0.571}_{+0.118}^{\pm 0.005}$	$0.446^{\pm 0.018}$	$0.360^{\pm 0.022}$
SWM (Ours)	${0.773}_{+0.052}^{\pm 0.008}$	$0.586^{\pm 0.017}$	$0.363^{\pm 0.021}$	${0.613}_{+0.160}^{\pm 0.006}$	$0.499^{\pm 0.015}$	$0.309^{\pm 0.018}$

Appendix

Table 5: Complete agent means and standard deviations for the main comparison. Values are means, with standard deviations in superscripts where available. Return subscripts show the mean difference from the base agent within each dataset. Recall measures affected-consumer coverage; broken services are counted per task.

Figure 6: Real-library prediction and executed exploration. (a) Blast-set F1 on 984 held-out records; hollow markers show F1 on the 137 graph-missed breaking records. Error bars retain the reported three-seed spreads; unmarked learned variants are single runs. (b) Prediction after adding random or blind-spot exploration, with base-corpus seed spreads. (c) Acquisition yields from 450 candidate perturbations per policy on three training worlds, conditional on passing local checks. Consumer AUROC measures discrimination among broken and unbroken consumers.

Interface	Extracted breaks	Silent
Set-valued, record-order context	0.417	0.841
Per-consumer, record-order context	0.000	1.000
Per-consumer, change-focused context	0.586	0.808
+ consumer usage	0.575	0.780
+ usage and blind-spot round	0.576	0.805
Alternative 1.5B backbone	0.689	0.799

Appendix

Table 6: Additional interface diagnostics on the same 984 held-out records. Columns retain the recorded “Extracted breaks” and “Silent” measures; blast-set and hidden-channel F1 appear in Figure 6 a.

Corpus	Records	Return	$\Delta$ return [95% interval]	Candidate recall	Broken services
Random	100	0.745	$+0.023\;[-0.008,+0.062]$	0.573	0.342
Random	200	0.767	$+0.046\;[+0.018,+0.082]$	0.594	0.342
Random	360	0.772	$+0.050\;[+0.021,+0.088]$	0.569	0.383
Coverage	360	0.768	$+0.046\;[+0.012,+0.089]$	0.573	0.375
Expert	360	0.780	$+0.059\;[+0.029,+0.095]$	0.603	0.342
Decision-aware	100	0.765	$+0.043\;[+0.013,+0.081]$	0.607	0.342

Appendix

Table 7: All corpus budgets in the agent comparison, averaged over three seeds. Intervals are paired world-bootstrap 95% intervals against base-agent return

0.721

Figure 7: Training budget and agent behavior on the same 40 tasks. Return gains use paired world-bootstrap 95% intervals; candidate recall and broken-service counts show three-seed half-ranges, with the base agent as the dashed reference.

Figure 8: Forecast accuracy depends on both propagation mechanism and input information. (a) Baselines on 90 held-out synthetic forecast steps; error bars show three-seed half-ranges, and unmarked learned variants are single diagnostic runs. (b) Direct and hidden-channel F1 across nine checkpoints, with 360, 315, and 819 seed-record pairs in the three groups, respectively. (c) Nested graph inputs under identical labels; the star increases the runtime input budget from 30 to 60. Oracle traversal restores hidden graph channels.

Learned configurations
Configuration	Seed 0	Seed 1	Seed 2	Mean $\pm$ half-range
text input, per-module clip	0.642	0.614	0.565	$0.607\pm 0.039$
graph tokens $+$ text input, per-module clip	0.621	0.549	0.579	$0.583\pm 0.036$
text input, shared clip (preregistered)	0.565	0.537	0.611	$\mathbf{0.571\pm 0.037}$
Structure
typed reachability	0.431	0.431	0.431	0.431

Appendix

Table 8: Per-module against shared gradient clipping for the companion recipe, three paired training seeds, forecast-step blast-set F1 on the

90

held-out synthetic forecast steps. The graph-token arm uses

K_{g}{=}8

learned attention queries and per-module clipping.

Figure 9: Explicit-belief decisions and prediction-error diagnostics. (a) Mean return for preventive-migration plans on 40 held-out tasks;

N

is the number of sampled blast sets. The learned

N=64

error bar is the three-seed standard error. (b) Return versus mean expenditure on paid checks: shape identifies the check strategy and filled markers denote calibrated beliefs; the displayed error bar is a standard error. (c) Noise-perturbed oracles matched to learned F1 have lower ranking regret but lower planning return. Ranking and planning use separate populations; lower regret and higher return are better.

World	Composition	Changed interface	Executed signal
Procedural systems	$4$ – $7$ services, shared library, datastore, events, configuration and access control	Nine change families over typed relations	Repository suites, service health, checks, and true blast sets
Acme	Four repositories with three Flask services and one shared library	API, dependency, database, event, configuration, flag and permission changes	Local suites plus end-to-end checks
Online Boutique	Ten repositories containing nine Go, Node and Python microservices	gRPC contracts, service wiring, data and latency changes	Repository tests, service health, direct and web journeys
Real-library worlds	Marshmallow with four dependents; OpenTelemetry semantic conventions with four instrumentation packages	Locally consistent upstream API and contract changes	Frozen upstream and downstream repository test suites

Appendix

Table 9: Software worlds used for exploration, training, and evaluation. Each change must pass the changed repository’s own checks before downstream outcomes are measured.

Layer	Kind	Meaning
System	repo	a version-controlled repository
	service	a deployable component
	library	a shared importable component
	database	a datastore instance
	table	a table or collection
	topic	an event stream

Appendix

Table 10: Node kinds. Every surface-layer node has exactly one owning system-layer node, which identifies the component that owns each interface.

Relation	src $\rightarrow$ dst	Recovered from	Consumer
contains	repo $\rightarrow$ service/library	repository layout
depends_on	repo/service $\rightarrow$ library	manifest, lockfile	✓
imports_symbol	service $\rightarrow$ symbol	import statements	✓
provides	service $\rightarrow$ endpoint	route declarations
calls_api	service $\rightarrow$ endpoint	client call sites	✓
publishes	service $\rightarrow$ topic	bus client calls

Appendix

Table 11: Edge relations. The consumer relations are those along which breakage propagates backwards, and are the ones the reachability baseline traverses.

Family	Method	Input and output	Role
Structural	Typed reachability	Extracted graph and changed target; reachable consumers	Reachability baseline
Supervised risk	Action-only logistic	Change kind and target count; consumer probabilities	Action-prior control
Supervised risk	Relation/action logistic	Action features, typed degrees, coverage and hops; consumer probabilities	Strong tabular control
Neural structure	Graph neural network (GNN) only	System-Surface Graph; blast set	Graph encoder without language generation
Generative	Zero-shot code models	Serialized state and change; parsed blast set	Task training against model scale
Bounds	Empty, true-graph traversal, and oracle consequence	No failure, inaccessible structure, or executed outcome	Lower and upper references

Appendix

Table 12: Comparison methods and their roles. Learned controls are fitted only on training worlds. Oracle rows use information unavailable at decision time.

Predictor configuration
input	text input (2 hops; $\leq 30$ edges)
graph mode	none (no graph tokens)
base	Qwen2.5-Coder, 0.5B Instruct
adapters	LoRA $r{=}16$ , $\alpha{=}32$
target modules	q, k, v, o projections
precision	bfloat16

Appendix

Table 13: Consequence-predictor configuration. The default serialises retrieved state as text; graph-conditioned variants are specified in Appendix D.13 .

Software Engineering

Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.

See all 270 papers

Towards Evaluation of Implicit Software World Models in Coding LLMs
Jun 25, 2026Egor Bogomolov, Yaroslav ZharovLLM Evaluation World Model Evaluation
1JetBrains Research
SWE-Shift: Evaluating Coding Agents in Evolving Workspaces
Oct 23, 2026Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7Coding Agents Software Engineering Benchmarks
MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University +2
Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
Aug 4, 2026Chenyu Wang, Yunbo Lyu, Junda He +4Software Engineering Agents AI Agent Monitoring
Singapore Management University · University of Alberta · Nanjing University of Science and Technology +1

Jun 25, 2026J/K move · Enter open · S save

Towards Evaluation of Implicit Software World Models in Coding LLMs

Egor Bogomolov, Yaroslav Zharov

1JetBrains Research

LLM Evaluation World Model Evaluation

Open paper AskPDF arXiv

Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.

Abstract

Software Engineering

Explore similar work

Towards Evaluation of Implicit Software World Models in Coding LLMs

SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks