cs.CLApr 10, 2026

Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL

Authors: Vishnu Murali, Anmol Gulati, Elias Lumer, Kevin Frank, Sindy Campagna, Vamse Kumar Subbiah

Organizations: Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Translating natural language into Jira Query Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based benchmark exists for mapping natural language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic retrieval tool that resolves natural-language mentions of categorical values through embedding-based similarity search. Among 9 frontier LLMs evaluated, single-pass models average only 43.4% execution accuracy on short natural-language queries, highlighting that text-to-JQL remains an open challenge. The agentic approach improves 7 of 9 models, with a 9.0% relative gain on the most linguistically challenging variant; in a controlled ablation isolating JiraAnchor, categorical-value accuracy rises from 48.7% to 71.7%, with component-field accuracy jumping from 16.9% to 66.2%. Our analysis identifies inherent semantic ambiguities, such as issue-type disambiguation and text-field selection, as the dominant failure modes rather than value-resolution errors, pointing to concrete directions for future work. We publicly release the benchmark, all agent transcripts, and evaluation code to support reproducibility.

Figures & tables

Figure 1: Naive single-turn approach (left) compared to Agentic Jackal (right). The naive approach generates JQL in a single pass without tool access. The agent iteratively refines queries using JiraSearch and JiraAnchor (both available in Experiments 1 and 2), looping until it produces a verified output. The JiraAnchor tool enables the agent to resolve ambiguous user queries by retrieving relevant unique values in a field using the live Jira instance.
Figure 2: Overview of the two evaluation datasets. Left: Jackal-1K, a 1,000-query stratified subset of the Jackal benchmark with four NL variants per query, used in Experiment 1. Right: The Field-Value Evaluation Set, a targeted set of 2-clause queries covering three categorical fields, used in Experiment 2.
Figure 3: JiraAnchor pipeline. Given a field name and natural language value mention, the tool fetches candidate values from the live Jira instance, computes embedding-based similarity scores, and returns top-10 matches ( K=10K{=}10 ) ranked by cosine similarity.
Figure 4: Average execution accuracy by query variant (9 models). Short NL shows the largest agentic gain (improvement of 3.9%), while Sem. Exact is near ceiling for both conditions (improvement of 2.3%).
ModelNaiveAgenticΔ\Delta
Gemini 3 Flash.628.710+.082
Gemini 3 Pro.641.671+.030
GPT-5 Mini.616.638+.022
Claude Opus 4.6.634.653+.019
Pixtral Large.537.554+.017
GPT-5.2.659.670+.011
Table 1: Overall execution accuracy, naive compared to agentic on Jackal-1K. Seven of nine models improve. Table 7 (Appendix) provides full per-variant detail.
Figure 5: Average execution accuracy by categorical field (9 models). JiraAnchor yields the largest gain on component queries (improvement of 49.3%), where baseline accuracy without value grounding collapses to 16.9%.
ModelBaselinew/ JiraAnchorΔ\Delta
GPT-5 Mini.393.710+.317
GPT-5.2.416.705+.289
Pixtral Large.453.742+.289
Claude Sonnet 4.5.462.733+.271
GPT-4o.449.710+.261
Gemini 3 Pro.527.742+.215
Table 2: Overall execution accuracy with and without JiraAnchor across 9 models. All models show positive gains. Table 8 (Appendix) provides full per-field detail.
NL MentionWithoutWith JiraAnchor
“7.0” field: fixVersion‘‘7.0’’ 0 results‘‘7.0 (Next Major Release)’’ correct
“build tools” field: component‘‘build tools’’ 0 results‘‘Build tools: Other’’ correct
“QDS 1.6 Beta” field: fixVersionnot resolved‘‘QDS 1.6 Beta’’ correct
Table 3: JiraAnchor success cases. Each row isolates a single field-value mention. Without JiraAnchor, the agent guesses a value that does not exist; with it, the canonical stored form is retrieved.
Figure 6: Error taxonomy for Agentic Jackal failures by query variant. Each bar shows the proportion of failures attributed to six error categories. On the hardest variants (Short NL and Sem. Similar), semantic interpretation errors dominate: Issue Type Interp. (e.g., “bugs” as issuetype=Bug vs. text search), Text Field Selection ( summary vs. description ), and Version Confusion ( affectedVersion vs. fixVersion ). These are inherent ambiguities that tools cannot resolve.
Table 10
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

ClausesJQLLong NLSem. ExactSem. SimilarShort NL
2updated <= -90d AND issuetype in ("Epic")I’m checking for epics that haven’t been updated in the last 90 days, so we can identify any long-standing items that might need attention or could potentially be closed.Updated is less than or equal to 90 days ago, and issue type is EpicLarge-scale tasks that haven’t been changed in the last three monthsEpics inactive for 90+ days
3created >= -4w AND assignee is EMPTY AND issuetype in ("User Story")I’m checking for any user stories that have been created in the last four weeks but haven’t been assigned to anyone yet, so we can make sure nothing important is slipping through the cracks.Created is within the last 4 weeks, assignee is empty, and issue type is User StoryStories added in the last month that haven’t been assigned to anyone yetUnassigned user stories created in last 4 weeks
4updated >= "2025-01-01" AND issuetype in ("Bug") AND priority is not EMPTY AND description ˜ "crash"I’m searching for bug reports that have been updated since the start of 2025, specifically those that mention a crash in their description and have a priority set. This helps me focus on recent and prioritized crash-related issues that might need urgent attention.Updated is on or after 2025-01-01, issue type is Bug, priority is not empty, and description contains crashProblems flagged as bugs, with a set urgency level, mentioning crashes, and modified after the start of 2025Bugs with priority and crash in description updated since 2025
5updated <= "2025-01-01" AND description ˜ "error" AND affectedVersion is not EMPTY AND resolution is EMPTY AND issuetype in ("Epic", "User Story", "Task", "Sub-task")I’m looking for any Epics, User Stories, Tasks, or Sub-tasks that mention ‘error’ in their description, have at least one affected version specified, haven’t been resolved yet, and were last updated on or before January 1st, 2025.Updated is on or before 2025-01-01, description contains error, affected version is not empty, resolution is empty, and issue type is Epic, User Story, Task, or Sub-taskItems that mention an error, haven’t been resolved yet, are linked to a specific version, and were last changed before January 2025, covering all major and minor work categories.Unresolved issues with error in description
Appendix
Table 6: JQL queries with increasing clause complexity, each paired with four natural language request variants.
 NaiveAgenticΔ\Delta (Agentic −- Naive)
ModelOverallOverallOverallLong NLShort NLSem. ExactSem. Similar
Gemini 3 Flash.628.710+.082+.067+.173+.059+.028
Gemini 3 Pro.641.671+.030−- .024+.091+.012+.039
GPT-5 Mini.616.638+.022−- .035+.063+.020+.040
Claude Opus 4.6.634.653+.019−- .035+.047+.020+.044
Pixtral Large.537.554+.017+.036−- .008+.126−- .087
Appendix
Table 7: Full per-variant execution accuracy, Experiment 1. Δ\Delta columns show the change (agentic −- naive) for each query variant. Bold marks the best value per column.
 Fix VersionAffected VersionComponentsOverall Avg.
ModelBase+JAΔ\DeltaBase+JAΔ\DeltaBase+JAΔ\DeltaBase+JAΔ\Delta
Claude Sonnet 4.5.639.736+.097.611.764+.153.137.699+.562.462.733+.271
Claude Opus 4.6.736.778+.042.722.778+.056.425.644+.219.628.733+.105
Gemini 3 Flash.790.710−- .080.806.871+.065.206.619+.413.601.733+.132
Gemini 3 Pro.667.722+.055.694.792+.098.219.712+.493.527.742+.215
GPT-4o.569.736+.167.667.750+.083.110.644+.534.449.710+.261
Appendix
Table 8: Full per-field execution accuracy, Experiment 2 (JiraAnchor ablation). Base = Baseline Agent (without JiraAnchor), +JA = Agent with JiraAnchor. Bold marks the best value per column.
More in this topic

Tool-Augmented Language Model Agents

Tool-augmented language model agents use external tools, APIs, and environments to solve tasks beyond text generation alone.

See all 167 papers

Explore similar work

CardsList
Sep 19, 2026

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL agent that closes this gap end to end: a six-node LangGraph StateGraph checks question relevance, fetches the live schema, generates PostgreSQL, validates it with a dry run, retries on failure, executes the query, and narrates the result set in plain English. The agent uses prompt engineering only; no model was fine-tuned. We evaluated it on the Chicago Crime dataset (approximately 8.5 million records, 22 attributes) against a hand-built benchmark of 100 natural language questions with ground-truth SQL, stratified into 30 Easy, 40 Medium and 30 Hard items. Comparing two prompt revisions of the same agent, the revised system (V2) reached a Valid SQL Rate of 93% (from 87%), an Execution Accuracy of 60% under a hybrid relational equivalence metric (from 47%; 19% from 12% under strict JSON matching), and a mean Synthesis Quality of 4.34 out of 5 (from 3.91). The single largest driver was removing a LIMIT 10 instruction from the system prompt, which had been truncating multi-row answers. Error analysis attributes the residual failures to relevance-checker false rejections, ambiguous question semantics, and free-tier API rate limits rather than to the language generation step. We report no comparison against an external baseline system or a public benchmark; the study is a single-model engineering evaluation.

Vigneshwar Ravi Rao, Rupesh Swarnakar, Fayeq Jeelani Syed†

Luddy School of Informatics, Computing and Engineering, Indiana University, Indianapolis, Indiana, USA

Text-to-SQLTool-Using Language AgentsText-to-SQL EvaluationLLM Agent Evaluation
AskPDF
Oct 14, 2026

J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While tau-bench is widely used to evaluate LLM agents, it is English-centric and cannot evaluate agents in other languages. Existing multilingual extensions offer no public Japanese version and rely primarily on machine translation, leaving locale-specific entities such as addresses and currencies inconsistent with the target region. To enable reliable evaluation in Japanese, we carefully localize tau-bench through machine translation, entity adaptation, and human review. During this process, we also uncover and fix two flaws in the original tau-bench: invalid tasks and a score aggregation error that inflates certain models' results. Experiments show that J-tau yields fewer evaluation errors than a naively machine-translated counterpart and that our fixes resolve both flaws.

Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa +1

SB Intuitions Corp. · Nagoya University

AI Agent BenchmarksLLM Agent EvaluationTool-Using Language AgentsCross-Cultural Language Model Evaluation
AskPDF
Apr 24, 2026

PExA: Parallel Exploration Agent for Complex Text-to-SQL

LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.

Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3

University of California Los Angeles · Bloomberg

LLM Inference EfficiencyText-to-SQLLLM Agents
AskPDF