Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL

Vishnu Murali; Anmol Gulati; Elias Lumer; Kevin Frank; Sindy Campagna; Vamse Kumar Subbiah

cs.CLApr 10, 2026

Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL

Authors: Vishnu Murali, Anmol Gulati, Elias Lumer, Kevin Frank, Sindy Campagna, Vamse Kumar Subbiah

Organizations: Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.

Abstract

Translating natural language into Jira Query Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based benchmark exists for mapping natural language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic retrieval tool that resolves natural-language mentions of categorical values through embedding-based similarity search. Among 9 frontier LLMs evaluated, single-pass models average only 43.4% execution accuracy on short natural-language queries, highlighting that text-to-JQL remains an open challenge. The agentic approach improves 7 of 9 models, with a 9.0% relative gain on the most linguistically challenging variant; in a controlled ablation isolating JiraAnchor, categorical-value accuracy rises from 48.7% to 71.7%, with component-field accuracy jumping from 16.9% to 66.2%. Our analysis identifies inherent semantic ambiguities, such as issue-type disambiguation and text-field selection, as the dominant failure modes rather than value-resolution errors, pointing to concrete directions for future work. We publicly release the benchmark, all agent transcripts, and evaluation code to support reproducibility.

Figures & tables

Figure 1: Naive single-turn approach (left) compared to Agentic Jackal (right). The naive approach generates JQL in a single pass without tool access. The agent iteratively refines queries using JiraSearch and JiraAnchor (both available in Experiments 1 and 2), looping until it produces a verified output. The JiraAnchor tool enables the agent to resolve ambiguous user queries by retrieving relevant unique values in a field using the live Jira instance.

Figure 2: Overview of the two evaluation datasets. Left: Jackal-1K, a 1,000-query stratified subset of the Jackal benchmark with four NL variants per query, used in Experiment 1. Right: The Field-Value Evaluation Set, a targeted set of 2-clause queries covering three categorical fields, used in Experiment 2.

Figure 3: JiraAnchor pipeline. Given a field name and natural language value mention, the tool fetches candidate values from the live Jira instance, computes embedding-based similarity scores, and returns top-10 matches ( $K{=}10$ ) ranked by cosine similarity.

Figure 4: Average execution accuracy by query variant (9 models). Short NL shows the largest agentic gain (improvement of 3.9%), while Sem. Exact is near ceiling for both conditions (improvement of 2.3%).

Model	Naive	Agentic	$\Delta$
Gemini 3 Flash	.628	.710	+.082
Gemini 3 Pro	.641	.671	+.030
GPT-5 Mini	.616	.638	+.022
Claude Opus 4.6	.634	.653	+.019
Pixtral Large	.537	.554	+.017
GPT-5.2	.659	.670	+.011

Table 1: Overall execution accuracy, naive compared to agentic on Jackal-1K. Seven of nine models improve. Table 7 (Appendix) provides full per-variant detail.

Figure 5: Average execution accuracy by categorical field (9 models). JiraAnchor yields the largest gain on component queries (improvement of 49.3%), where baseline accuracy without value grounding collapses to 16.9%.

Model	Baseline	w/ JiraAnchor	$\Delta$
GPT-5 Mini	.393	.710	+.317
GPT-5.2	.416	.705	+.289
Pixtral Large	.453	.742	+.289
Claude Sonnet 4.5	.462	.733	+.271
GPT-4o	.449	.710	+.261
Gemini 3 Pro	.527	.742	+.215

Table 2: Overall execution accuracy with and without JiraAnchor across 9 models. All models show positive gains. Table 8 (Appendix) provides full per-field detail.

NL Mention	Without	With JiraAnchor
“7.0” field: fixVersion	‘‘7.0’’ 0 results	‘‘7.0 (Next Major Release)’’ correct
“build tools” field: component	‘‘build tools’’ 0 results	‘‘Build tools: Other’’ correct
“QDS 1.6 Beta” field: fixVersion	not resolved	‘‘QDS 1.6 Beta’’ correct

Table 3: JiraAnchor success cases. Each row isolates a single field-value mention. Without JiraAnchor, the agent guesses a value that does not exist; with it, the canonical stored form is retrieved.

Figure 6: Error taxonomy for Agentic Jackal failures by query variant. Each bar shows the proportion of failures attributed to six error categories. On the hardest variants (Short NL and Sem. Similar), semantic interpretation errors dominate: Issue Type Interp. (e.g., “bugs” as issuetype=Bug vs. text search), Text Field Selection ( summary vs. description ), and Version Confusion ( affectedVersion vs. fixVersion ). These are inherent ambiguities that tools cannot resolve.

Table 10

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Clauses	JQL	Long NL	Sem. Exact	Sem. Similar	Short NL
2	updated <= -90d AND issuetype in ("Epic")	I’m checking for epics that haven’t been updated in the last 90 days, so we can identify any long-standing items that might need attention or could potentially be closed.	Updated is less than or equal to 90 days ago, and issue type is Epic	Large-scale tasks that haven’t been changed in the last three months	Epics inactive for 90+ days
3	created >= -4w AND assignee is EMPTY AND issuetype in ("User Story")	I’m checking for any user stories that have been created in the last four weeks but haven’t been assigned to anyone yet, so we can make sure nothing important is slipping through the cracks.	Created is within the last 4 weeks, assignee is empty, and issue type is User Story	Stories added in the last month that haven’t been assigned to anyone yet	Unassigned user stories created in last 4 weeks
4	updated >= "2025-01-01" AND issuetype in ("Bug") AND priority is not EMPTY AND description ˜ "crash"	I’m searching for bug reports that have been updated since the start of 2025, specifically those that mention a crash in their description and have a priority set. This helps me focus on recent and prioritized crash-related issues that might need urgent attention.	Updated is on or after 2025-01-01, issue type is Bug, priority is not empty, and description contains crash	Problems flagged as bugs, with a set urgency level, mentioning crashes, and modified after the start of 2025	Bugs with priority and crash in description updated since 2025
5	updated <= "2025-01-01" AND description ˜ "error" AND affectedVersion is not EMPTY AND resolution is EMPTY AND issuetype in ("Epic", "User Story", "Task", "Sub-task")	I’m looking for any Epics, User Stories, Tasks, or Sub-tasks that mention ‘error’ in their description, have at least one affected version specified, haven’t been resolved yet, and were last updated on or before January 1st, 2025.	Updated is on or before 2025-01-01, description contains error, affected version is not empty, resolution is empty, and issue type is Epic, User Story, Task, or Sub-task	Items that mention an error, haven’t been resolved yet, are linked to a specific version, and were last changed before January 2025, covering all major and minor work categories.	Unresolved issues with error in description

Appendix

Table 6: JQL queries with increasing clause complexity, each paired with four natural language request variants.

	Naive	Agentic	$\Delta$ (Agentic $-$ Naive)
Model	Overall	Overall	Overall	Long NL	Short NL	Sem. Exact	Sem. Similar
Gemini 3 Flash	.628	.710	+.082	+.067	+.173	+.059	+.028
Gemini 3 Pro	.641	.671	+.030	$-$ .024	+.091	+.012	+.039
GPT-5 Mini	.616	.638	+.022	$-$ .035	+.063	+.020	+.040
Claude Opus 4.6	.634	.653	+.019	$-$ .035	+.047	+.020	+.044
Pixtral Large	.537	.554	+.017	+.036	$-$ .008	+.126	$-$ .087

Appendix

Table 7: Full per-variant execution accuracy, Experiment 1.

\Delta

columns show the change (agentic

-

naive) for each query variant. Bold marks the best value per column.

	Fix Version			Affected Version			Components			Overall Avg.
Model	Base	+JA	$\Delta$	Base	+JA	$\Delta$	Base	+JA	$\Delta$	Base	+JA	$\Delta$
Claude Sonnet 4.5	.639	.736	+.097	.611	.764	+.153	.137	.699	+.562	.462	.733	+.271
Claude Opus 4.6	.736	.778	+.042	.722	.778	+.056	.425	.644	+.219	.628	.733	+.105
Gemini 3 Flash	.790	.710	$-$ .080	.806	.871	+.065	.206	.619	+.413	.601	.733	+.132
Gemini 3 Pro	.667	.722	+.055	.694	.792	+.098	.219	.712	+.493	.527	.742	+.215
GPT-4o	.569	.736	+.167	.667	.750	+.083	.110	.644	+.534	.449	.710	+.261

Appendix

Table 8: Full per-field execution accuracy, Experiment 2 (JiraAnchor ablation). Base = Baseline Agent (without JiraAnchor), +JA = Agent with JiraAnchor. Bold marks the best value per column.

Tool-Augmented Language Model Agents

Tool-augmented language model agents use external tools, APIs, and environments to solve tasks beyond text generation alone.

See all 167 papers

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database
Sep 19, 2026Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†Text-to-SQL Tool-Using Language Agents
Luddy School of Informatics, Computing and Engineering, Indiana University, Indianapolis, Indiana, USA
J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction
Oct 14, 2026Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa +1AI Agent Benchmarks LLM Agent Evaluation
SB Intuitions Corp. · Nagoya University
PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg

Sep 19, 2026J/K move · Enter open · S save

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†

Luddy School of Informatics, Computing and Engineering, Indiana University, Indianapolis, Indiana, USA

Text-to-SQL Tool-Using Language Agents Text-to-SQL Evaluation LLM Agent Evaluation

Open paper AskPDF arXiv

Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL agent that closes this gap end to end: a six-node LangGraph StateGraph checks question relevance, fetches the live schema, generates PostgreSQL, validates it with a dry run, retries on failure, executes the query, and narrates the result set in plain English. The agent uses prompt engineering only; no model was fine-tuned. We evaluated it on the Chicago Crime dataset (approximately 8.5 million records, 22 attributes) against a hand-built benchmark of 100 natural language questions with ground-truth SQL, stratified into 30 Easy, 40 Medium and 30 Hard items. Comparing two prompt revisions of the same agent, the revised system (V2) reached a Valid SQL Rate of 93% (from 87%), an Execution Accuracy of 60% under a hybrid relational equivalence metric (from 47%; 19% from 12% under strict JSON matching), and a mean Synthesis Quality of 4.34 out of 5 (from 3.91). The single largest driver was removing a LIMIT 10 instruction from the system prompt, which had been truncating multi-row answers. Error analysis attributes the residual failures to relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than to the language generation step. We report no comparison against an external baseline system or a public benchmark; the study is a single-model engineering evaluation.

Abstract

Tool-Augmented Language Model Agents

Explore similar work

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

PExA: Parallel Exploration Agent for Complex Text-to-SQL