SUMMIR: A Hallucination-Aware Framework for Ranking Sports Insights from LLMs

Nitish Kumar; Sannu Kumar; S Akash; Manish Gupta; Ankith Karat; Sriparna Saha

cs.IRMar 30, 2026

SUMMIR: A Hallucination-Aware Framework for Ranking Sports Insights from LLMs

Authors: Nitish Kumar, Sannu Kumar, S Akash, Manish Gupta, Ankith Karat, Sriparna Saha

Organizations: Indian Institute of Technology Patna, India · Microsoft, India

Abstract

With the rapid proliferation of online sports journalism, extracting meaningful pre-game and post-game insights from articles is essential for enhancing user engagement and comprehension. In this paper, we address the task of automatically extracting such insights from articles published before and after matches. We curate a dataset of 7,900 news articles covering 800 matches across four major sports: Cricket, Soccer, Basketball, and Baseball. To ensure contextual relevance, we employ a two-step validation pipeline leveraging both open-source and proprietary large language models (LLMs). We then utilize multiple state-of-the-art LLMs (GPT-4o, Qwen2.5-72B-Instruct, Llama-3.3-70B-Instruct, and Mixtral-8x7B-Instruct-v0.1) to generate comprehensive insights. The factual accuracy of these outputs is rigorously assessed using a FactScore-based methodology, complemented by hallucination detection via the SummaC (Summary Consistency) framework with GPT-4o. Finally, we propose SUMMIR (Sentence Unified Multimetric Model for Importance Ranking), a novel architecture designed to rank insights based on user-specific interests. Our results demonstrate the effectiveness of this approach in generating high-quality, relevant insights, while also revealing significant differences in factual consistency and interestingness across LLMs. This work contributes a robust framework for automated, reliable insight generation from sports news content. The source code is availble here https://github.com/nitish-iitp/SUMMIR.

Figures & tables

New Records	(1) India posted the highest-ever first innings score in a T20 World Cup final with 176 runs for the loss of seven wickets.
Key Match Events	(1) Virat Kohli scored an inspired 76 runs off 59 balls. (2) Jasprit Bumrah bowled two magnificent overs, conceding only six runs in the 16th and 18th overs. (3) Suryakumar Yadav took an otherworldly catch to dismiss David Miller. (4) Heinrich Klaasen struck a magnificent 23-ball half-century. (5) Hardik Pandya took 3 wickets for 20 runs. (6) India won the T20 World Cup by seven runs. (7) Bumrah castled Marco Jansen and conceded only two runs in a crucial over. (8) Arshdeep Singh bowled an economical penultimate over, conceding only four runs. (9) Suryakumar Yadav’s catch in the final over was pivotal in sealing the victory.
Post-Match Reflections	(1) Jasprit Bumrah expressed his joy and pride after the match, stating, ‘We play the sport for this, I am really over the moon.’ (2) Virat Kohli reflected on his performance and the significance of the win, saying, ‘This was my last T20 World Cup, and this is what we wanted to achieve.’ (3) Proteas captain Aiden Markram expressed his disappointment but pride in the team’s performance, stating, ‘It hurts quite a bit, but I’m really proud of the team.
Misc. Highlights	(1) India claimed their second T20 World Cup, 17 years after winning their first. (2) The match was held at Kensington Oval in Bridgetown, Barbados.

Table 1: Sample insights generated for Mens Cricket match between India vs South Africa T20I World Cup Final 2024

Figure 1: Structured pipeline for news filtering, LLM-based insight generation, and performance evaluation in building the sports insights dataset

Model	Precision (%)	Recall (%)	F1-score (%)
Falcon 10B [ 2 ]	81.90	80.10	81.00
Qwen 2.5 14B [ 48 ]	84.90	91.50	88.07
Mistral Nemo-12.2B [ 22 ]	82.30	88.10	85.09
Llama 3.1 8B [ 15 ]	79.10	91.30	84.75
Phi-4 14B [ 1 ]	86.00	80.80	83.30
Llama 3.3 70B (4-bit quant.) [ 15 ]	81.90	95.10	88.02

Table 2: Comparative performance of open-source LLMs on 996 manually labeled articles for match relevance validation

Panel of Figure 2 — Figure 2: Structured prompt template used for extracting relevance and categorized insights from Cricket match articles.

Sport	Fact-Score (%)				SummaC (%)
	Llama 70B	Mixtral- 8x7B	Qwen 72B	GPT- 4o	Llama 70B	Mixtral- 8x7B	Qwen 72B	GPT- 4o
Soccer	94	92	93	95	58	52	56	60
Basketball	95	91	93	97	55	50	58	68
Baseball	95	88	93	96	58	53	56	69
Cricket	94	94	93	95	69	63	64	72

Table 3: Evaluation of 4 LLMs on Fact-Score and SummaC metrics across 4 sports

Figure 3: Insight ranking framework integrating diverse scoring features, PPO-based LLM fine-tuning, and evaluation metrics including NDCG, Recall, and SUMMIR (Sentence Unified Multimetric Model for Importance Ranking).

Reward/Metric	NDCG@2	NDCG@5	NDCG@10	Recall@2	Recall@5	Recall@10
NDCG	0.865	0.866	0.911	0.700	0.680	0.920
Recall	0.855	0.839	0.910	0.700	0.560	0.860
SUMMIR	0.858	0.908	0.943	0.800	0.760	0.960

Table 4: Evaluation of PPO-based fine-tuning of Llama 3.2 1B using NDCG, Recall, and SUMMIR as reward signals. SUMMIR uses ScoreNet-based rankings instead of gold rankings with LLM-generated permutations.

1. Australia will now shift their focus to South Africa and the Eden Gardens.
2. The win ensures Australia will enter Thursday’s semi-final against South Africa as one of the competition’s form teams.
3. For Bangladesh, the defeat marks the end of a dour campaign, but they have ensured they get to the Champions Trophy.
4. Australia lost their first two games but bounced back to win seven on the trot to seal a spot in the semi-finals.
5. Mitch Marsh’s performance will please Australia most.

Table 5: SUMMIR generated rankings of insights for “43rd match of the ICC Cricket World Cup 2023, between Australia and Bangladesh on Nov 11, 2023, at Pune, India.”

Learning to Rank

Learning to rank develops models that order items by relevance or utility for search, recommendations, and other decisions.

See all 266 papers

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports
Jul 27, 2026Jonas Schröder, Jonas Schweisthal, Oliver Müller +2LLM Evaluation Forecasting Benchmarks
MCML & LMU Munich Munich, Germany · Paderborn University Paderborn, Germany · University of Cologne Cologne, Germany
Tree-of-Text: A Tree-based Prompting Framework for Table-to-Text Generation in the Sports Domain
Apr 29, 2026Shang-Hsuan Chiang, Tsan-Tsung Yang, An-Zi Yen +1Grounded Text Generation Sports Analytics
National Yang Ming Chiao Tung University
A Survey of Large Models in Sports
Aug 14, 2026Yichen Xu, Jianzhe Ma, Chuhan Wang +4Sports Analytics Multimodal Large Language Models
Renmin University of China · Sichuan University · Beijing University of Posts and Telecommunications

Jul 27, 2026J/K move · Enter open · S save

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

Jonas Schröder, Jonas Schweisthal, Oliver Müller, Markus Weinmann+1

MCML & LMU Munich Munich, Germany · Paderborn University Paderborn, Germany · University of Cologne Cologne, Germany

LLM Evaluation Forecasting Benchmarks Probabilistic Forecasting Sports Outcome Prediction

Open paper AskPDF arXiv

Large language models (LLMs) increasingly support decisions about uncertain future events, yet evaluating their ability to forecast real-world outcomes remains difficult. In particular, existing benchmarks are typically static and retrospective, and therefore cannot test how information is synthesized by LLMs to predict future events under uncertainty. We introduce LLM-SoccerArena (https://llm-soccerarena.com), a prospective live benchmark that evaluates how well LLMs forecast real-world sports events before the outcomes are known. LLM-SoccerArena provides (1) a prospective live benchmark protocol, (2) a public open-source platform, and (3) a factorial benchmark design together with tournament-related questions (e.g., which team will win). LLM-SoccerArena automatically records timestamped, schema-validated forecasts of unresolved events, together with prompts, model versions, tool traces, and costs. The factorial design varies along four dimensions: (1) model version (e.g., GPT-5.5, Claude Opus 4.8); (2) information access; (3) prompting strategy, and (4) forecast horizon. We demonstrate LLM-SoccerArena through a large-scale evaluation of the 2026 FIFA World Cup, in which seven LLMs generated forecasts for all 104 matches and 15 tournament-related questions. We provide a detailed analysis of model performance across information access, prompting strategy, and forecast horizon. As a result, LLM-SoccerArena provides new evidence about the forecasting performance of state-of-the-art LLMs. For example, LLMs with web access outperform those without, but only by a small margin (i.e., a 0.023 improvement in Brier score). Overall, LLM-SoccerArena provides a flexible, open-source platform for prospective benchmarking of unresolved events. LLM-SoccerArena will be continuously updated, and can be directly applied to future national and international tournaments and league competitions.

Abstract

Learning to Rank

Explore similar work

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

Tree-of-Text: A Tree-based Prompting Framework for Table-to-Text Generation in the Sports Domain

A Survey of Large Models in Sports