CivicBench: A Benchmark and Memory Framework for Web-Based Civic Data Analytics

Yangyu Wu; Siru Zhong; Haoyuan Liang; Yuxuan Liang

cs.AIOct 31, 2026

CivicBench: A Benchmark and Memory Framework for Web-Based Civic Data Analytics

Authors: Yangyu Wu, Siru Zhong, Haoyuan Liang, Yuxuan Liang

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Tsinghua University, Beijing, China

Abstract

Municipalities publish service requests, geographic boundaries, weather observations, demographic estimates, and operational guidance through independently maintained Web portals. Civic analysis must combine these sources while preserving their schemas, source links, spatial and temporal scopes, and interpretive limits. Tool-using language agents can perform this integration, but existing task benchmarks rarely test whether a computed claim remains supported by the records used. We present CivicBench, a 400-task benchmark over four cities and two years that materializes civic Web data into frozen bundles for reproducible evaluation. Executable contracts score computation, source linkage, coverage, scope, qualified answers, and refusal when the requested conclusion is not supported. We also introduce Task-State Memory (TSM), which compiles tool observations into typed assertions, reconciles records only within compatible scopes, and retrieves a bounded state organized around unresolved constraints. Across nine completed model pairs, eight yield positive mean-score differences with 95% semantic-contract-cluster bootstrap intervals excluding zero; the ninth interval crosses zero. The benchmark and memory framework make the records behind a claim inspectable at the point of decision.

Figures & tables

Figure 1. Benchmark scope and task composition. Six public evidence sources cover four cities and two years; each of the 400 tasks combines a computation with provenance, scope, and a claim boundary. Six civic Web evidence sources feed a benchmark of 400 tasks across four cities, two years, ten capabilities, and three answer types.

Figure 2. Benchmark construction pipeline. Evidence-derived targets and contracts are fixed before governance-facing questions are framed. Public tasks, evaluator ground truth, and internal generation state remain separate, while target/scorer, operation, and no-record shortcut checks gate the frozen artifact. A five-stage pipeline acquires public civic data, constructs source-native evidence, derives targets and contracts, frames tasks, and checks the frozen benchmark.

City	Municipal, boundary, and NOAA sources	311 rows
Austin	Austin Open Data 311 ( xwdj-i9he ); council districts ( w3v2-cj58 ); Camp Mabry ( USW00013958 )	280,395
Boston	Analyze Boston 311; BPDA neighborhood boundaries; Logan ( USW00014739 )	324,124
Chicago	Chicago Data Portal 311 ( v6vf-nfxy ); community areas ( igwz-8jzy ); O’Hare ( USW00094846 )	445,933
San Francisco	SF Open Data 311 ( vw6y-z8j6 ); supervisor districts ( hcgx-vtsb ); Downtown ( USW00023272 )	1,454,388
Total		2,504,840

Table 1. Civic Web sources and two-year service-record snapshots.

Band	Capability	Required evidence operation	Expected output	Answer	$n$
D1	Data fitness	Check schema, manifest scope, and category presence	Field- and file-level evidence	Direct	24
D2	Priority ranking	Aggregate a scoped metric; rank eligible units	Ranked units with metric and coverage	Direct	30
	Threshold screening	Align dates; build event/baseline windows; threshold units	Selected units with window statistics	Direct	44
	Data quality	Audit coordinate coverage and sampled boundary fit	Spatial-risk ranking and citywide accounting	Direct	24
D3	Priority ranking	Aggregate a scoped metric; rank eligible units	Ranked units with metric and coverage	Qualified	52
	Trend change	Align 2024–2025 records; compute signed changes	Ranked changes with interpretation boundary	Qualified	46

Table 2. Composition of the 400-task benchmark by evidence operation and answer type. D3 contributes 254 tasks that require explicit qualification.

Figure 3. Public task and hidden evaluation. The agent receives the question, visible contract, and declared source files. Scoring type, oracle provenance, source traces, and evaluator state remain hidden until prediction is complete. The public task exposes a question, contract, and source files to the agent, while the scoring contract, oracle provenance, and evaluator remain hidden.

Figure 4. Task-State Memory. INIT compiles the public task; WRITE, RESOLVE, and CHECK maintain typed records and constraint state; RETRIEVE forms a bounded projection; and DECIDE uses that context to continue, answer, qualify, or refuse. A six-stage memory loop compiles a task, writes and resolves typed observations, checks constraints, retrieves a bounded projection, and supports the model's next decision.

Frozen reference runs
	Mean task score $\uparrow$			Exact match $\uparrow$		Completion $\uparrow$
Model	ReAct	TSM	$\Delta$ [95% CI]	ReAct	TSM	ReAct	TSM
DeepSeek V4 Flash	0.2782	0.3451	+0.0669 [0.0336, 0.1008]	0.0000	0.0050	0.9425	0.9500
Grok 4 Fast Reasoning	0.2502	0.3102	+0.0600 [0.0263, 0.0936]	0.0100	0.0150	0.9950	0.9700
Additional endpoints
GPT-5.4-nano	0.1373	0.2089	+0.0717 [0.0473, 0.0962]	0.0025	0.0000	0.9950	0.9550

Table 3. RQ1 effectiveness across nine model pairs; eight TSM intervals exclude zero.

Variant	Score $\uparrow$	$\Delta$	Exact $\uparrow$
Full TSM	0.3621	–	0.005
w/o Resolve	0.3605	-0.0016	0.000
w/o Check	0.3499	-0.0122	0.000
w/o Retrieve	0.2343	-0.1278	0.000
w/o Decide	0.3398	-0.0223	0.005

Table 4. RQ2 TSM stage ablations on the frozen DeepSeek suite. Removing Retrieve causes the largest score loss.

(a) Assertion-level trace checks
Trace check	DeepSeek	Grok
Tool-result trace	1432/1432	1303/1303
File-read trace	1426/1429	1302/1302
Field trace	1333/1333	1255/1255
Spatial-scope match	555/555	404/404
Temporal-scope match	802/802	631/631

Table 5. RQ3 trace validity and task-evidence coverage.

Model	Complete $\uparrow$ $\Delta$ (tasks)	Failures $\downarrow$ $\Delta$	Tokens $\downarrow$ $\Delta$ /task	Latency $\downarrow$ $\Delta$ (s/task)
DeepSeek V4	+3	-15	-6,555	+4.94
Grok 4 Fast	-10	-41	+515	+5.09

Table 6. RQ4 operational changes from ReAct to TSM.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Scorer and shortcut controls
Audit	Overall	Worst subgroup
Oracle/scorer regression	1.0000	1.0000
Always refuse	0.0983	0.2000
Always empty	0.1177	0.2000
Keyword stuffing	0.0891	0.2000
Majority answer shape	0.0986	0.2038

Appendix

Table 7. Benchmark integrity and no-evidence shortcut audits. Evidence-free strategies reach at most 0.1715 overall task score.

Computation, synthesis, and quality
	DeepSeek		Grok
Metric	ReAct	TSM	ReAct	TSM
Rank	0.4390	0.5198	0.3818	0.4535
Trend	0.3799	0.7937	0.2799	0.5855
Threshold	0.1403	0.1687	0.1472	0.1264
Group	0.1314	0.1437	0.1167	0.1489

Appendix

Table 8. RQ1 endpoint scores by capability.

City
	DeepSeek		Grok
Metric	ReAct	TSM	ReAct	TSM
Austin	0.3171	0.3589	0.3207	0.4010
Boston	0.2188	0.2745	0.1516	0.2057
Chicago	0.2794	0.3624	0.2873	0.3074
San Francisco	0.3117	0.4034	0.2603	0.3498

Appendix

Table 9. RQ1 endpoint scores by city, answer type, and evidence-role complexity.

		Valid/eligible assertion traces					Task coverage
Model	Method	Parent	File	Field	Spatial	Temporal	Role cov.	Open constraint
DeepSeek	TSM	1432/1432	1426/1429	1333/1333	555/555	802/802	0.9044	150/158
	Raw	untyped	untyped	untyped	untyped	untyped	0.8395	untyped
Grok	TSM	1303/1303	1302/1302	1255/1255	404/404	631/631	0.7373	249/249
	Raw	untyped	untyped	untyped	untyped	untyped	0.6982	untyped

Appendix

Table 10. RQ3 evidence-state trace validity and coverage.

			Outcome change			Resource change
Model	Case	Task	Score $\Delta$	Structured $\Delta$	Boundary $\Delta$	Tokens $\Delta$	Latency $\Delta$
DeepSeek	Structured fix	ug_23eba3fdf29f	+0.8500	+1.0000	+0.3333	-10,953	-4.31
	Regression	ug_72c901dcb068	-0.8500	-1.0000	-0.3333	+14,602	+21.65
Grok	Scope fix	ug_42fdd9bc9fd7	+0.9075	+1.0000	+0.4167	+4,284	+2.33
	Regression	ug_5b5a2f57011f	-0.9125	-1.0000	-0.4167	+22,325	+15.65

Appendix

Table 11. RQ4 paired corrections and regressions with observable cost changes.

Failures and answer diagnostics
	DeepSeek		Grok
Metric	ReAct	TSM	ReAct	TSM
Complete	377	380	398	388
API	1	0	2	10
Max step	9	10	0	0
Malformed	13	10	0	2

Appendix

Table 12. RQ4 endpoint reliability diagnostics and resource use.

Metric	DeepSeek	Grok
Main subset $\Delta$	+0.0515[-0.0409, 0.1453]	+0.0517[-0.0631, 0.1663]
Replication $\Delta$	+0.0615[-0.0200, 0.1452]	+0.1414[0.0363, 0.2497]
Rep.–main	+0.0100[-0.1084, 0.1304]	+0.0897[-0.0333, 0.2316]
Task- $\Delta$ $r$	0.076	0.257
MAE	0.2688	0.2436

Appendix

Table 13. Endpoint stability on the pre-frozen 40-task replication subset.

Data Analysis Agents

Data analysis agents use language models and tools to explore datasets, run analyses, and produce answers or reports grounded in evidence.

See all 114 papers

AgenticCity-Bench: Longitudinal Multimodal Observations of Urban Dynamics with Agentic Reasoning
Oct 26, 2026Xiaofei Yu, Kailai Sun, Yanyu Mao +7Multimodal Dataset Agent Benchmarks
Renmin University of China · Singapore-MIT Alliance for Research and Technology, MIT · Massachusetts Institute of Technology +4
DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?
Apr 10, 2026Young-Suk Lee, Ramon Fernandez Astudillo, Radu FlorianAI Agent Benchmarks Scientific Agent Benchmarks
IBM Research
CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information
Sep 8, 2026Dingying Liu, Yunshun Zhong, Wentao Zhang +1Agent Failure Analysis AI Agent Reliability
University of Sydney · University of Toronto · University of Waterloo +1

Oct 26, 2026J/K move · Enter open · S save

AgenticCity-Bench: Longitudinal Multimodal Observations of Urban Dynamics with Agentic Reasoning

Xiaofei Yu, Kailai Sun, Yanyu Mao, Baoshen Guo+6

Renmin University of China · Singapore-MIT Alliance for Research and Technology, MIT · Massachusetts Institute of Technology+4

Multimodal Dataset Agent Benchmarks Multimodal Spatial Reasoning Urban Planning Spatiotemporal Forecasting

Open paper AskPDF arXiv

City systems are shaped by interactions among infrastructure, human activity, and environmental conditions. Existing earth observation and urban benchmarks are limited in unified modeling and in capturing their joint dynamics. We introduce AgenticCity-Bench, a longitudinal multimodal benchmark for multi-layer unified state modeling and agentic city task reasoning. The benchmark covers 16 cities in 12 countries across six continents from 2000 to 2025, and includes 18 modalities spanning urban function, activity, and environmental state. It organizes heterogeneous data sources around a common spatial reference and calendar index while preserving spatial and temporal coverage, provenance and tile-level availability. On this foundation, AgenticCity-Bench supports urban cross-modal translation together with diverse agentic city tasks involving forecasting, planning and resource analysis. This study provides a unified benchmark for studying the transition from multimodal urban and earth observation to joint understanding and agentic city decision-making. The dataset is available at https://anonymous.4open.science/r/AgenticCity-Bench.

Abstract

Data Analysis Agents

Explore similar work

AgenticCity-Bench: Longitudinal Multimodal Observations of Urban Dynamics with Agentic Reasoning

DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information