CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence

Yuzhe Zhang; Weijie Zhu; Haolin Yang; Ziyun Zhang; Xianwei Xue; Mengke Chen; Qiutong Pan; Huaqian Cai

cs.CLSep 29, 2026

CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence

Authors: Yuzhe Zhang, Weijie Zhu, Haolin Yang, Ziyun Zhang, Xianwei Xue, Mengke Chen, Qiutong Pan, Huaqian Cai

Organizations: Peking University · National Key Lab of Data Space Technology and System · Baidu Inc.

Abstract

Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work. We introduce CypherTurn, the first benchmark for conversational Text-to-Cypher evaluation, comprising 721 sessions and 5,927 turns across 7 knowledge graphs and 13 conversational phenomena. We evaluate 15 models under a guided oracle protocol and a fully autonomous agentic protocol, yielding four findings. First, the best model reaches only 64.7% execution accuracy, and session-level correctness remains below 5%. Second, despite strong overall rank correlation, frontier models exhibit a consequential reordering of the top of the leaderboard under autonomous operation, a phenomenon we term the Autonomy Divergence, which reveals error-management as a partially independent capability from raw generation skill. Third, scaling action budgets from x3 to x10 fails to close the autonomy gap, as the strongest frontier models self-limit to approximately two actions per turn regardless of available budget. Fourth, single-turn Cypher fine-tuning degrades multi-turn instruction following, while architecture-appropriate specialization outperforms several frontier models. These results establish CypherTurn as an open challenge for conversational graph database reasoning. Code and data are available at https://github.com/BarryQ/CypherTurn.

Figures & tables

Figure 1: Existing single-turn benchmarks evaluate each query in isolation (left); CypherTurn evaluates chain-dependent turns in sequence, where each query must be grounded in its predecessor’s result (right).

Benchmark	MT	Cypher	Ctx	Agent	T/S
CypherBench	✗	✓	✗	✗	1.0
ZOGRASCOPE	✗	✓	✗	✗	1.0
SM3-Text-to-Query	✗	✓	✗	✗	1.0
Text2GQL-Bench	✗	✓	✗	✗	1.0
MTGQL †	✓	✗	✗	✗	6.5
CypherTurn	✓	✓	✓	✓	8.2

Table 1: Comparison of CypherTurn against existing Text-to-Cypher and graph query benchmarks. MT : multi-turn sessions; Cypher : Neo4j Cypher target; Ctx : contextual anaphora evaluation; Agent : agentic protocol; T/S : average turns per session. † MTGQL targets nGQL, not Cypher.

Figure 2: Benchmark composition. (a) Phenomenon distribution across 5,206 phenomenon-bearing turns (721 first turns carry no phenomenon; note: the First phenomenon refers to rank-then-select navigation, not to first turns): inner ring shows four categories (Navigation, Aggregation, Filtering, Discourse); outer ring shows individual phenomena. (b) Chain mode distribution across all 5,927 turns. (c) Persona distribution across 721 sessions; ten types are sampled near-uniformly.

Figure 3: The two evaluation protocols in CypherTurn . The Guided Protocol (left) supplies gold-annotated prior turns as oracle context, isolating raw Cypher generation ability. The Agentic Protocol (right) requires models to autonomously accumulate their own history under a configurable shared session budget ( $B=T\times m$ ); early-turn errors propagate forward through chain dependencies, compounding across the session.

	Guided Protocol					Agentic $\times$ 3
Model	EX $\uparrow$	PSJS $\uparrow$	CER $\downarrow$	SEM $\uparrow$	Tok/S	EX $\uparrow$	PSJS $\uparrow$	CER $\downarrow$	SEM $\uparrow$	Tok/S	$\Delta$ EX $\downarrow$
Frontier Models
Claude Opus 4.7	0.647	0.846	0.434	0.046	617	0.402	0.681	0.712	0.019	639	$-$ 0.245
GPT-5.5	0.625	0.828	0.424	0.025	504	0.401	0.673	0.698	0.006	554	$-$ 0.224
Kimi-K2.5	0.602	0.819	0.469	0.028	753	0.335	0.612	0.749	0.006	965	$-$ 0.267
Gemini-3.1-Flash-Lite	0.572	0.744	0.508	0.017	500	0.432	0.698	0.695	0.008	427	$-$ 0.140

Table 2: Turn-level results under the Guided and Agentic

\times

3 protocols. Tok/S : mean clean output tokens per session (excl. <think> ). Bold : best per column (excluding

\dagger

\dagger

CypherRI-7B’s near-zero EX and inflated token count both reflect a format compliance failure caused by fill-in-the-middle token contamination in its RL training; excluded from aggregate statistics.

Figure 4: Agentic EX under

\times

3 /

\times

5 /

\times

10 action budgets for 14 evaluated models. Full numeric tables in Appendix E.5 .

Figure 5: Guided EX across three dimensions for 14 models (CypherRI-7B excluded; MiniMax-M2.7 additionally excluded from heatmaps). (a) Per-phenomenon mean EX with min–max range; dashed line shows the unweighted macro mean across 13 phenomena. (b) Per-persona heatmap (13 models $\times$ 10 personas). (c) Per-graph heatmap (13 models $\times$ 7 graphs).

Figure 6: EX by turn position for six frontier models. (a) Guided protocol: all models remain stable across turns. (b) Agentic protocol: Claude, Gemini, and GLM-5 plateau at non-trivial EX (dashed lines show their guided baselines). (c) Agentic protocol: ERNIE, DeepSeek, and Qwen3 collapse to near-zero by

T_{10}

Model	INSPECT%	Agentic EX	$\Delta$ EX
GPT-5.5	94.1	0.401	0.224
Claude Opus 4.7	85.8	0.402	0.245
Gemini-3.1-FL	83.5	0.432	0.140
GLM-5	54.3	0.274	0.232
STRuCT-LLM-Novo	40.4	0.333	0.193
Kimi-K2.5	34.8	0.335	0.267

Table 3: INSPECT_SCHEMA fraction vs. Agentic

\times

3 EX, sorted by INSPECT%. Green : schema-heavy; yellow : mixed; red : schema-sparse.

Model	Paradigm	Guided	Agentic
Gemma-2-9B	Base	0.281	0.136
text-to-cypher-gemma	Single-turn SFT	0.261	0.143
QwQ-32B ‡	Base	0.395	0.324
STRuCT-LLM-Novo	RL + CoT	0.526	0.333

Table 4: Two controlled base-to-specialised pairs. Single-turn SFT degrades the Gemma pair, while RL with chain-of-thought improves the QwQ pair, in opposite directions. ‡ QwQ-32B is evaluated on the stratified 210-session subset of Appendix F , whose per-model EX deviates from the full set by at most 0.01; the 13.1-point gain far exceeds this subset noise.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Schema overview of the seven CypherTurn knowledge graphs. Each row: graph name, topology diagram, and node/relation listing. (a)–(d): directed cycle, bipartite cycle, diamond convergence, star+lateral; (e)–(g): citation self-loop, hierarchical self-loop, hub-and-spoke.

Phenomenon	Category	Chain Mode	Dep.	Definition and Cypher Pattern
EXPAND	Navigation	expand	✓	Traverse one hop outward from the prior result set; the prior result acts as the seed node set. MATCH (n)-[:R]->(m) WHERE n.name IN $prev
PIVOT	Navigation	pivot	✓	Rank the prior result by a property, then traverse from the top- $k$ entities. WITH n ORDER BY n.prop DESC LIMIT k MATCH (n)-[:R]->(m)
FIRST	Navigation	pivot	✓	Reference the single top-ranked entity from the prior result and traverse from it. WITH n ORDER BY n.prop DESC LIMIT 1 MATCH (n)-[:R]->(m)
TOPIC_SHIFT	Discourse	topic_shift	✗	A context-break: the user poses an entirely new, independent query unrelated to prior results. The only non-chain-dependent phenomenon.
AGG_AVG	Aggregation	agg_numeric	✓	Compute the mean of a numeric property over the prior result set. MATCH (n) WHERE n.name IN $prev RETURN avg(n.prop)
AGG_MAX	Aggregation	agg_numeric	✓	Find the maximum value of a numeric property in the prior result set. MATCH (n) WHERE n.name IN $prev RETURN max(n.prop)

Appendix

Table 5: The 13 conversational phenomena in CypherTurn . Dep. : chain-dependent (✓) or independent (✗). Chain Mode = the session construction mode used to generate turns of this phenomenon.

Chain Mode	Dep.	Definition and Example
None (T1)	✗	First turn of a session; no prior result exists. The query is fully self-contained. E.g. “List all provinces.” $\to$ MATCH (p:Province) RETURN p.name
topic_shift	✗	A deliberate context-break mid-session. The user abandons the prior thread and opens a new independent query. E.g. (after querying officials) “Actually, list all trade routes.”
expand	✓	Traverse one hop outward from the prior result. The prior entity set is used directly as a seed. E.g. (after T1 provinces) “What cities are in those provinces?”
pivot	✓	Rank the prior result by a property, select top- $k$ , and traverse from those entities. E.g. “Among those cities, which one has the highest population? Show its legions.”
agg_numeric	✓	Compute a numeric aggregate (avg, max, sum) over a property of the prior result set. E.g. “What is the average tribute gold of those provinces?”
aggregate	✓	Count the cardinality of the prior result set. E.g. “How many legions were returned above?”

Appendix

Table 6: The 9 chain modes in CypherTurn . Dep. : chain-dependent (✓) or independent (✗). None (T1) and topic_shift are the two non-chain-dependent modes; all others require the prior result to formulate the current query.

Persona	Definition	Example Utterance
Journalist	Investigative, direct phrasing; seeks evidence and facts; often frames queries as questions with implicit causal intent.	“Which provinces experienced the sharpest decline in tribute payments last year?”
Student	Exploratory and learning-oriented; may ask elementary questions or seek clarification on results.	“I’m trying to understand the hierarchy — can you show me which officials answer to the Emperor?”
Product Manager	Goal-focused; wants summary statistics and ranked lists; minimal tolerance for schema details.	“Give me the top three trade routes by cargo volume.”
Novice User	Vague phrasing; often uses pronouns ambiguously (“those things”, “that one”) without clear referents.	“Now what about the other ones? The ones with the big numbers?”
Data Analyst	Precise and technical; requests specific columns, sorted output, and numeric breakdowns.	“Return the entity ID, name, and tribute_gold for each province, ordered descending.”
Casual User	Informal register; contractions, colloquialisms, and incomplete sentences. May mix social and query intent.	“ok cool — so like, how many of those are there actually?”

Appendix

Table 7: The 10 session personas in CypherTurn . Each session is assigned one persona uniformly at random (66–77 sessions per type across 721 sessions total). Persona governs utterance register for every turn in the session.

Criterion	$\kappa$
Gold Cypher captures intent	0.91
Cross-turn anaphora unambiguous	0.86
Utterance natural for assigned persona	0.64
Pooled (reported)	0.83

Appendix

Table 8: Inter-annotator agreement

\kappa

decomposed by criterion. Persona naturalness is the weakest criterion. Criterion-level values are a retrospective decomposition based on the annotation protocol structure; the pooled

\kappa=0.83

is measured.

Action	Input	Returns
EXECUTE_CYPHER	Cypher query string	Result rows; error on failure
INSPECT_SCHEMA	Empty (ignored)	Node labels, properties, relation types
SEARCH_VALUES	JSON: label, property, query	Up to 10 matching values
ASK_USER	Natural language question	String reply from user
SUBMIT_ANSWER	Final Cypher query	"Answer submitted"

Appendix

Table 9: The five-action space shared by both evaluation protocols. Each action costs 1 budget unit.

Frontier Models
Model	Anc	Mag	Ocn	Stl	Arc	Cel	Mer	Avg
Claude Opus 4.7	0.674	0.594	0.660	0.603	0.683	0.663	0.650	0.647
GPT-5.5	0.633	0.570	0.659	0.597	0.671	0.667	0.581	0.625
Kimi-K2.5	0.620	0.568	0.588	0.551	0.667	0.618	0.605	0.602
Gemini-3.1-Flash-Lite	0.574	0.620	0.535	0.520	0.604	0.595	0.555	0.572
Qwen3-235B	0.548	0.488	0.529	0.472	0.535	0.564	0.457	0.513

Appendix

Table 10: Guided EX by knowledge graph. Anc=Ancient Empire, Mag=Magic Academy, Ocn=Ocean Kingdom, Stl=Stellar Colony, Arc=Arcane Archive, Cel=Celestial Court, Mer=Merchant Harbor. Bold : per-row maximum.

\dagger

excluded from mean row.

Frontier Models
Model	Anc	Mag	Ocn	Stl	Arc	Cel	Mer	Avg
Claude Opus 4.7	0.427	0.376	0.404	0.358	0.410	0.454	0.384	0.402
GPT-5.5	0.433	0.323	0.494	0.389	0.416	0.415	0.340	0.401
Kimi-K2.5	0.363	0.275	0.339	0.278	0.373	0.411	0.311	0.335
Gemini-3.1-Flash-Lite	0.445	0.479	0.442	0.392	0.409	0.446	0.409	0.432
Qwen3-235B	0.221	0.167	0.206	0.166	0.184	0.248	0.204	0.199

Appendix

Table 11: Agentic

\times

3 EX by knowledge graph. Same layout as Table 10 . Bold : per-row maximum.

\dagger

excluded from mean row.

Phenom.	Cat.	Cl	GPT	Km	Gfl	ST	Q3	GL	DS	ER	Mi	Mx	Ll	G2	t2	Mean
AGG_MAX	Agg	0.842	0.813	0.788	0.741	0.796	0.798	0.840	0.793	0.603	0.810	0.310	0.360	0.409	0.197	0.650
AGG_AVG	Agg	0.767	0.679	0.740	0.677	0.740	0.722	0.780	0.700	0.578	0.702	0.285	0.336	0.422	0.298	0.602
TOPIC_SHIFT	Disc	0.821	0.849	0.760	0.782	0.680	0.643	0.487	0.656	0.778	0.422	0.111	0.788	0.538	0.673	0.642
AGG_SUM	Agg	0.642	0.663	0.661	0.655	0.665	0.692	0.696	0.676	0.547	0.661	0.281	0.331	0.397	0.318	0.563
VALUE_FILTER	Filt	0.725	0.753	0.764	0.590	0.657	0.691	0.562	0.702	0.590	0.528	0.112	0.270	0.354	0.382	0.549
COUNT	Agg	0.670	0.614	0.636	0.583	0.514	0.657	0.673	0.604	0.520	0.611	0.274	0.156	0.352	0.240	0.507

Appendix

Table 12: Guided EX by phenomenon for all 14 models (CypherRI excluded). Cl=Claude 4.7, GPT=GPT-5.5, Km=Kimi-K2.5, Gfl=Gemini-3.1-FL, ST=STRuCT-LLM, Q3=Qwen3-235B, GL=GLM-5, DS=DeepSeek-V3.2, ER=ERNIE-5.0, Mi=MiMo-V2.5-Pro, Mx=MiniMax-M2.7, Ll=Llama-3.1-8B, G2=Gemma-2-9B, t2=t2c-gemma. Red rows : mean EX below 0.30. Bold : best per row.

Phenom.	Cat.	Cl	GPT	Km	Gfl	ST	Q3	GL	DS	ER	Mi	Mx	Ll	G2	t2	Mean
AGG_MAX	Agg	0.663	0.611	0.599	0.579	0.549	0.505	0.579	0.510	0.443	0.601	0.480	0.195	0.187	0.121	0.473
AGG_AVG	Agg	0.576	0.500	0.507	0.455	0.471	0.374	0.516	0.381	0.316	0.527	0.399	0.141	0.155	0.130	0.389
TOPIC_SHIFT	Disc	0.533	0.644	0.433	0.739	0.446	0.225	0.289	0.233	0.563	0.249	0.222	0.178	0.200	0.313	0.376
AGG_SUM	Agg	0.524	0.501	0.457	0.418	0.414	0.345	0.426	0.343	0.310	0.443	0.331	0.121	0.119	0.094	0.346
VALUE_FILTER	Filt	0.399	0.416	0.225	0.337	0.287	0.101	0.169	0.112	0.292	0.112	0.039	0.090	0.135	0.197	0.208
COUNT	Agg	0.548	0.455	0.308	0.246	0.333	0.109	0.402	0.100	0.062	0.324	0.146	0.000	0.003	0.000	0.217

Appendix

Table 13: Agentic

\times

3 EX by phenomenon for all 14 models (CypherRI excluded). Same abbreviations as Table 12 . Red rows : mean EX below 0.10. Bold : best per row.

Chain Mode	Dep.	Cl	GPT	Km	Gfl	ST	Q3	GL	DS	ER	Mi	Mx	Ll	G2	t2	Mean
None (T1)	✗	0.519	0.447	0.533	0.619	0.338	0.207	0.229	0.212	0.498	0.132	0.049	0.523	0.368	0.351	0.359
topic_shift	✗	0.821	0.849	0.760	0.782	0.680	0.643	0.487	0.656	0.778	0.422	0.111	0.788	0.538	0.673	0.642
agg_numeric	✓	0.753	0.716	0.730	0.690	0.733	0.737	0.771	0.723	0.576	0.724	0.292	0.342	0.409	0.271	0.603
aggregate	✓	0.670	0.614	0.636	0.583	0.514	0.657	0.673	0.604	0.520	0.611	0.274	0.156	0.352	0.240	0.507
value_narrow	✓	0.650	0.655	0.652	0.473	0.572	0.500	0.441	0.500	0.433	0.382	0.072	0.193	0.332	0.305	0.440
pivot	✓	0.611	0.554	0.489	0.513	0.401	0.501	0.456	0.494	0.335	0.416	0.068	0.083	0.094	0.103	0.366

Appendix

Table 14: Guided EX by chain mode for all 14 models (CypherRI excluded). Same model abbreviations as Table 12 . Bold : best per row. Dep. : chain-dependent (✓) or independent (✗).

Chain Mode	Dep.	Cl	GPT	Km	Gfl	ST	Q3	GL	DS	ER	Mi	Mx	Ll	G2	t2	Mean
None (T1)	✗	0.383	0.409	0.304	0.555	0.347	0.202	0.211	0.214	0.510	0.164	0.136	0.455	0.411	0.411	0.336
topic_shift	✗	0.533	0.644	0.433	0.739	0.446	0.225	0.289	0.233	0.563	0.249	0.222	0.178	0.200	0.313	0.376
agg_numeric	✓	0.584	0.534	0.517	0.479	0.474	0.404	0.503	0.407	0.353	0.519	0.399	0.150	0.152	0.114	0.399
aggregate	✓	0.548	0.455	0.308	0.246	0.333	0.109	0.402	0.100	0.062	0.324	0.146	0.000	0.003	0.000	0.217
value_narrow	✓	0.326	0.307	0.198	0.262	0.233	0.075	0.115	0.086	0.251	0.075	0.029	0.083	0.118	0.158	0.165
pivot	✓	0.321	0.246	0.294	0.424	0.259	0.147	0.227	0.149	0.209	0.203	0.037	0.055	0.055	0.053	0.191

Appendix

Table 15: Agentic

\times

3 EX by chain mode for all 14 models (CypherRI excluded). Same abbreviations as Table 14 . Bold : best per row.

Frontier Models
Model	Imp	Jou	Cas	Stu	PMg	NNS	DA	Nov	Adv	Vrb	Avg
Claude Opus 4.7	0.559	0.627	0.678	0.661	0.621	0.635	0.619	0.646	0.708	0.720	0.647
GPT-5.5	0.569	0.624	0.652	0.646	0.622	0.629	0.603	0.602	0.611	0.698	0.625
Kimi-K2.5	0.595	0.588	0.624	0.609	0.585	0.621	0.593	0.573	0.569	0.678	0.602
Gemini-3.1-Flash-Lite	0.516	0.574	0.591	0.552	0.553	0.567	0.555	0.561	0.583	0.676	0.572
Qwen3-235B	0.507	0.505	0.526	0.465	0.509	0.498	0.543	0.468	0.542	0.578	0.513

Appendix

Table 16: Guided EX by persona for all 14 models (CypherRI excluded). Imp=Impatient Executive, Jou=Journalist, Cas=Casual User, Stu=Student, PMg=Product Manager, NNS=Non-Native Speaker, DA=Data Analyst, Nov=Novice User, Adv=Adversarial User, Vrb=Verbose User. Bold : per-row maximum.

Frontier Models
Model	Imp	Jou	Cas	Stu	PMg	NNS	DA	Nov	Adv	Vrb	Avg
Claude Opus 4.7	0.298	0.411	0.423	0.383	0.397	0.319	0.384	0.417	0.470	0.519	0.402
GPT-5.5	0.347	0.409	0.441	0.394	0.397	0.363	0.382	0.396	0.447	0.437	0.401
Kimi-K2.5	0.295	0.331	0.378	0.324	0.319	0.328	0.379	0.273	0.363	0.369	0.335
Gemini-3.1-Flash-Lite	0.383	0.423	0.444	0.427	0.399	0.404	0.437	0.481	0.424	0.495	0.432
Qwen3-235B	0.145	0.207	0.214	0.160	0.195	0.147	0.268	0.193	0.208	0.254	0.199

Appendix

Table 17: Agentic

\times

3 EX by persona for all 14 models (CypherRI excluded). Imp=Impatient Executive, Jou=Journalist, Cas=Casual User, Stu=Student, PMg=Product Manager, NNS=Non-Native Speaker, DA=Data Analyst, Nov=Novice User, Adv=Adversarial User, Vrb=Verbose User. Bold : per-row maximum.

	Agentic $\times$ 3					Agentic $\times$ 5					Agentic $\times$ 10
Model	EX	PSJS	CER	SEM	Tok/S	EX	PSJS	CER	SEM	Tok/S	EX	PSJS	CER	SEM	Tok/S
Frontier Models
Claude Opus 4.7	0.402	0.681	0.712	0.019	639	0.429	0.714	0.697	0.025	620	0.424	0.706	0.704	0.019	634
GPT-5.5	0.401	0.673	0.698	0.006	554	0.395	0.661	0.707	0.004	564	0.406	0.680	0.704	0.007	546
Kimi-K2.5	0.335	0.612	0.749	0.006	965	0.346	0.621	0.734	0.008	1,006	0.355	0.628	0.731	0.012	934
Gemini-3.1-Flash-Lite	0.432	0.698	0.695	0.008	427	0.431	0.697	0.694	0.007	427	0.430	0.695	0.694	0.007	404

Appendix

Table 18: Agentic Protocol results across three action budgets (

\times

3 /

\times

5 /

\times

10). Tok/S : mean clean output tokens per session (excl. <think> ). — : results unavailable.

\dagger

excluded from aggregate statistics.

Frontier Models
Model	EXEC	INSPECT	SEARCH	ASK
Claude Opus 4.7	0.048	0.858	0.001	0.093
GPT-5.5	0.011	0.941	0.002	0.046
Gemini-3.1-Flash-Lite	0.107	0.835	0.002	0.056
Kimi-K2.5	0.631	0.348	0.002	0.019
GLM-5	0.407	0.543	0.001	0.048

Appendix

Table 19: Agentic

\times

3 action usage fractions (excluding SUBMIT_ANSWER ). EXEC= EXECUTE_CYPHER , INSPECT= INSPECT_SCHEMA , SEARCH= SEARCH_VALUES , ASK= ASK_USER . Models that allocate over 80% to INSPECT show the smallest agentic degradation; those allocating over 67% to EXEC show the largest cascade failures.

Figure 8: Chain Error Rate (CER) by turn position under the Guided (solid) and Agentic

\times

3 (dashed) protocols for five representative models.

Variant	$\rho$	Models	Top-1	Gemini keeps top-1
Agentic-PS	0.915	10	GPT-5.5 (0.463)	No
Disclosed-Horizon	0.879	10	Claude (0.456)	No
Budget-Encouraging	0.786	7	GPT-5.5 (0.399)	No

Appendix

Table 20: Summary of the three protocol variants.

\rho

is the Spearman rank correlation between the base agentic ranking and the variant ranking over the same model set. The Budget-Encouraging variant is restricted to seven valid models because three models suffered API failures that corrupted their trajectories.

Model	Base	Agentic-PS	$\Delta$
GPT-5.5	0.402	0.463	$+$ 0.061
Claude Opus 4.7	0.400	0.461	$+$ 0.061
Kimi-K2.5	0.327	0.426	$+$ 0.099
Gemini-3.1-Flash-Lite	0.433	0.435	$+$ 0.002
GLM-5	0.281	0.316	$+$ 0.035
DeepSeek-V3.2	0.207	0.237	$+$ 0.030

Appendix

Table 21: Agentic-PS (persistent schema) per-model EX on the ablation subset. Bold marks the variant top-1. † MiMo-V2.5-Pro recovered only 53% of real turns before an API quota interruption, so its PS value is unreliable and it is excluded from the top-1 conclusion.

Model	Base	Disclosed-Horizon	$\Delta$
Claude Opus 4.7	0.400	0.456	$+$ 0.056
GPT-5.5	0.402	0.395	$-$ 0.008
Gemini-3.1-Flash-Lite	0.433	0.364	$-$ 0.069
Kimi-K2.5	0.327	0.317	$-$ 0.010
ERNIE-5.0	0.297	0.273	$-$ 0.024
GLM-5	0.281	0.252	$-$ 0.028

Appendix

Table 22: Disclosed-Horizon per-model EX on the ablation subset. Bold marks the variant top-1. Disclosing the horizon helps under-planning models such as Claude and DeepSeek-V3.2 and slightly hurts Gemini-3.1-Flash-Lite, whose schema-inspection strategy is disrupted by the extra planning signal.

Model	Base	BEP $\times$ 10	$\Delta$
GPT-5.5	0.402	0.399	$-$ 0.003
Claude Opus 4.7	0.400	0.367	$-$ 0.033
Gemini-3.1-Flash-Lite	0.433	0.354	$-$ 0.079
Kimi-K2.5	0.327	0.291	$-$ 0.036
GLM-5	0.281	0.200	$-$ 0.081
DeepSeek-V3.2	0.207	0.198	$-$ 0.010

Appendix

Table 23: Budget-Encouraging Prompt at

\times

10 budget, per-model EX on the ablation subset. Bold marks the variant top-1. Six of seven valid models decline; only Qwen3-235B, which had been under-planning, rises. ‡ These three models suffered complete API request failures during the run, with zero input tokens and no model output, so their values reflect quota contamination rather than genuine performance and are excluded from the rank correlation and the decline count.

Model	Base gap	Agentic-PS	DH	BEP
Gemini-3.1-Flash-Lite	$-$ 0.140	$-$ 0.137	$-$ 0.208	$-$ 0.218
Claude Opus 4.7	$-$ 0.245	$-$ 0.186	$-$ 0.191	$-$ 0.280
Qwen3-235B	$-$ 0.314	$-$ 0.292	$-$ 0.316	$-$ 0.285
DeepSeek-V3.2	$-$ 0.302	$-$ 0.269	$-$ 0.260	$-$ 0.308
Frontier spread	0.174	0.155	0.125	0.090

Appendix

Table 24: Guided-to-variant gap, measured as variant EX minus guided EX, for four frontier models. A value near zero would mean the variant closes the autonomy gap. No model approaches zero under any variant, and the frontier spread remains substantial throughout.

Model	Exhaust. (%)	EX (orig.)	EX (excl. cliff)	$\Delta$
Claude Opus 4.7	0.0	0.402	0.402	0.000
GPT-5.5	0.0	0.401	0.401	0.000
Gemini-3.1-Flash-Lite	0.3	0.432	0.432	0.000
Kimi-K2.5	26.1	0.335	0.358	$+$ 0.023
Qwen3-235B	62.6	0.199	0.231	$+$ 0.032
DeepSeek-V3.2	62.7	0.204	0.236	$+$ 0.033

Appendix

Table 25: Budget cliff decomposition. Exhaustion rate is the fraction of sessions whose shared budget reached zero. EX (excl. cliff) recomputes accuracy after removing turns auto-scored zero from budget exhaustion. The frontier is barely affected, while small models gain substantially.

Panel of Figure 9 — Figure 9: Case Study L.1: Claude Opus 4.7 guided session 49eb5670 . EX = 1.0 across all 8 turns (Celestial Court, adversarial persona) in 867 total clean tokens. Left: T1–T4; Right: T5–T8. Analysis below.

Panel of Figure 10 — Figure 10: Case Study L.2: DeepSeek-V3.2 agentic session 93c94edc . SQL/Cypher language confusion at T3 triggers a metacognitive failure; T4 exhausts the shared session budget (13 of 24 remaining actions, 363 tokens); T5–T8 receive zero actions. EX = 0.25, CER = 1.00, 604 total clean tokens.

Panel of Figure 11 — Figure 11: Case Study L.3: Gemini-3.1-Flash-Lite agentic session 97729ed3 . Strict INSPECT $\to$ SUBMIT strategy achieves EX = 0.778 (7/9 turns) with 349 total tokens (Ocean Kingdom, impatient_exec). Left: T1–T5; Right: T6–T9. Failures at T4 and T9 arise from context-chain loss; recovery at T5 is immediate.

Panel of Figure 12 — Figure 12: Case Study L.4: text-to-cypher-gemma guided session 3edd37d1 . Single-turn fine-tuning causes the “hardcoding anti-pattern” (T3, T6) and relation direction errors (T4, T7). EX = 0.375 (3/8 turns); 862 tokens total (Ancient Empire, casual_user). Left: T1–T4; Right: T5–T8.

Panel of Figure 13 — Figure 13: Case Study L.5: Llama-3.1-8B guided session f1e1e06b . SQL syntax contamination at T3 triggers cascading failure; T5 and T8 submit no Cypher despite oracle context; T9 reverts to citizen names from T1–T2. EX = 0.222 (2/9 turns); 1,220 total tokens (Ancient Empire, journalist). Left: T1–T5; Right: T6–T9.

Explore similar work

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems
Jun 7, 2026Suraj Ranganath, Anish RaghavendraText2Cypher Schema
Halıcıoğlu School of Data Science and Computing University of California, San Diego · Independent Researcher
Toward Multi-Database Query Reasoning for Text2Cypher
May 11, 2026Makbule Gulcin OzsoyText2Cypher
Neo4j, London, UK
CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment
Jun 3, 2026Nikodem TomczakSchema Validation
Thulge Labs, Singapore

Jun 7, 2026, cs.LGJ/K move · Enter open · S save

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Suraj Ranganath, Anish Raghavendra

Halıcıoğlu School of Data Science and Computing University of California, San Diego · Independent Researcher

Text2Cypher Schema

Open paper AskPDF arXiv

Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevant Text2Cypher benchmark therefore reflects the questions users and agents actually ask of that graph. Creating such a benchmark is difficult because schemas and values are unique, and graph structure changes over time. Each NL-query pair must also be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. We present PIPE-Cypher, a local benchmark-generation pipeline that turns a live property graph and optional seed queries from customer questions, analyst logs, or agent tool calls into balanced NL-to-Cypher benchmarks. PIPE-Cypher combines schema profiling, reverse-query grounding, constrained generation, deterministic Cypher governance, execution validation, redaction, diversity controls, and a calibrated local LLM judge. Using local Qwen3.5-9B generation and judging, PIPE-Cypher exports 3,000 accepted FinBench/SNB examples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models. The resulting benchmark is deliberately discriminative: zero-shot transfer is weak, while a few-shot control shows that schema-specific example banks can help compatible model families. Together, PIPE-Cypher makes Text2Cypher benchmarking a repeatable process that evolves with the graph, its users, and its target workloads.

Abstract

Explore similar work

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Toward Multi-Database Query Reasoning for Text2Cypher

CYGNET: Cypher Gate for Neural Execution Triage and Cost Containment