GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search

Sahajpreet Singh; Kokil Jaidka; Min-Yen Kan; Preslav Nakov

cs.CLFeb 9, 2026

GitSearch: Enhancing Community Notes Generation with Gap-Informed Targeted Search

Authors: Sahajpreet Singh, Kokil Jaidka, Min-Yen Kan, Preslav Nakov

Organizations: National University of Singapore

Abstract

Community Notes enable scalable, community-based fact-checking, but automated generation faces a coverage--quality trade-off. Note-dependent methods (e.g., summarization-based Supernotes) cannot operate without prior crowd input, while web-search LLMs achieve broad coverage but often retrieve evidence misaligned with what raters need. We introduce GitSearch (Gap-Informed Targeted Search), which treats missing, ambiguous, or disputed information as first-class retrieval targets. It infers information gaps from a post and optional incomplete notes, searches specifically to resolve them, and synthesizes an evidence-grounded Community Note. To evaluate this, we introduce PolBench, a new dataset of 78,698 U.S. political tweets with Community Notes. Keeping single-call search inference budget fixed, GitSearch improves human-rated helpfulness by 0.43 over unconditioned web search and wins 59% of pairwise comparisons. It generates notes for 99% of test cases, including the 46% where summarization methods fail. Human evaluators prefer GitSearch to human-written helpful notes in 69% of comparisons, primarily due to better context coverage. These findings show that how search is organized matters more than information access alone.

Figures & tables

Figure 1: Example of human-written and GitSearch -generated notes addressing misleading tweet.

Figure 2: Overview of our three-phase (in black numbering) GitSearch framework.

		LLM-as-a-Judge
Type	Model	FE $\bm{\uparrow}$	CA $\bm{\uparrow}$	FA $\bm{\uparrow}$	C $\bm{\uparrow}$	H $\bm{\uparrow}$
	Llama-3.1-8B	4.087	3.585	2.479	2.464	2.796
	Ministral-8B	4.083	3.589	2.487	2.355	2.672
	Qwen3-14B	3.974	3.728	2.615	2.487	2.838
	Apriel-Nemotron-15B	3.879	3.709	2.540	2.528	2.811
	GPT-5-nano	4.053	3.668	2.634	2.506	2.860

Table 1: LLM-as-a-Judge results for AI-generated CNs, using human-written helpful notes as ground truth. Here, Type represents generation method (coverage/success rate) and Scores (1–5): Functional Errors (FE), Claim Alignment (CA), Fact Alignment (FA), Completeness (C), and Helpfulness (H). Gray-shaded rows report differences (

\Delta

); green / red coloring is shown only for GitSearch comparisons. Statistical significance (using independent t-test), where ,,,

p<\{0.05,0.01,0.001\}

, respectively.

Method	F1 $\bm{\uparrow}$	F2 $\bm{\uparrow}$	F3 $\bm{\uparrow}$	F4 $\bm{\uparrow}$	C1 $\bm{\uparrow}$	C2 $\bm{\uparrow}$	H $\bm{\uparrow}$
Human	0.94	0.91	0.87	0.81	0.93	0.69	3.36
Supernote Lite	0.96	0.83	0.73	0.77	0.91	0.53	3.14
Web Agent	0.96	0.80	0.89	0.88	0.85	0.65	3.44
GitSearch	0.99	0.89	0.91	0.90	0.97	0.87	3.87
$\Delta_{\text{OUR}-\text{Human}}$	0.05	-0.02	0.04	0.09	0.04	0.18**	0.51**

Table 2: Human evaluation of AI-generated Community Notes. Factuality: F1 (factual accuracy), F2 (source accessibility), F3 (source support), F4 (source credibility). Completeness: C1 (relevance), C2 (context coverage). H: Helpfulness (1–5 scale).

Method	FE $\bm{\uparrow}$	CA $\bm{\uparrow}$	FA $\bm{\uparrow}$	C $\bm{\uparrow}$	H $\bm{\uparrow}$
Web Agent ( $\mathcal{C}_{i}=\emptyset$ )	4.475	3.345	2.457	2.435	2.991
GitSearch ( $\mathcal{C}_{i}=\emptyset$ )	4.238	3.578	2.565	2.623	3.049
${}^{\mathcal{C}_{i}=\emptyset}\Delta_{OUR-WA}$	-0.237 ***	0.233	0.108	0.188	0.058
Supernote Lite ( $\mathcal{C}_{i}\neq\emptyset$ )	4.053	3.668	2.634	2.506	2.86
GitSearch ( $\mathcal{C}_{i}\neq\emptyset$ )	4.291	3.743	2.732	2.826	3.219
${}^{\mathcal{C}_{i}\neq\emptyset}\Delta_{OUR-SN}$	0.238 ***	0.075	0.098	0.32 **	0.359 ***

Table 3: Ablation study quantifying the impact of including existing community notes (

\mathcal{C}_{i}

) as context.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	US Politics			Others			Macro-F1
Method	P	R	F1	P	R	F	Macro-F1
MNLI	0.925	0.428	0.585	0.911	0.994	0.951	0.768
Keywords	0.987	0.524	0.685	0.925	0.999	0.961	0.823
Hybrid	0.945	0.710	0.811	0.953	0.993	0.973	0.892

Appendix

Table 4: CNs data filteration pipeline’s quality check on random 1K samples.

Category	Statistic	Value(s)
Overall size	Number of tweets with notes	78,698
	Number of CNs	169,992
	Time span of tweets	2021-01-05 – 2025-09-06
	Time span of CNs	2021-01-28 – 2025-09-06
	Language(s)	English
Contributors	Unique contributors	4,288

Appendix

Table 5: Descriptive statistics of the PolBench .

Panel of Figure 6 — Figure 6: Prompt template for the Supernote-lite baseline. The model is instructed to synthesize a single “Super Community Note” by aggregating key points from existing notes, weighted by their helpfulness scores, without performing new external retrieval.

Panel of Figure 7 — Figure 7: Prompt template for the Web-agent baseline. Unlike the gap-driven approach, this agent is given a general instruction to use web search tools to verify the tweet and generate a helpful note in a single stage.

Panel of Figure 8 — Figure 8: Prompt template for GitSearch phase I (Gap Identification). The model functions as a detective to analyze the tweet and categorize specific information deficits (e.g., Unsubstantiated Claim, Missing Context) into a structured JSON list of prioritized search queries.

Panel of Figure 9 — Figure 9: Prompt template for GitSearch phase II (Targeted Search). The agent uses the prioritized gaps from Phase I to guide targeted web retrieval, synthesizing the retrieved evidence into an intermediate, authoritative fact-checking article.

Panel of Figure 10 — Figure 10: Prompt template for GitSearch phase III (CN Synthesis). The model is instructed to distill the comprehensive fact-checking article into a concise, platform-compliant note (max 280 characters) that adheres to neutrality and citation standards.

Panel of Figure 11 — Figure 11: Prompt template for the LLM-as-a-Judge evaluation. The evaluator is provided with strict rubric definitions to score generated notes on five dimensions: Functional Errors, Claim Alignment, Fact Alignment, Completeness, and Helpfulness.

Type (Generation Parameters)	Model Name	Detailed Model Name	Tool Used (If Needed)
Open-source LLM	Llama-3.1-8B Grattafiori et al. (2024)	meta-llama/Llama-3.1-8B-Instruct	–
(do_sample=False)	Ministral-8B Mistral-AI (2024)	mistralai/Ministral-8B-Instruct-2410	–
Open-source LRM	Apriel-Nemotron-15b Radhakrishna et al. (2025)	ServiceNow-AI/Apriel-Nemotron-15b-Thinker	–
(do_sample=False)	Qwen3-14B Yang et al. (2025)	Qwen/Qwen3-14B	–
Closed-source LRM	GPT-5-nano Singh et al. (2026)	gpt-5-nano-2025-08-07	Web search
(Default)	Gemini-2.5-Flash Comanici et al. (2025)	gemini-2.5-flash	Web search

Appendix

Table 6: An overview of models used in experiments. Note that LLM and LRM are used as short forms for Large Language Model and Large Reasoning Model, respectively.

Panel of Figure 12 — Figure 12: The taxonomy of questions presented to annotators. The evaluation task is divided into four categories: Factuality (F1–F4), Completeness (C1–C2), Helpfulness (H), and Ranking (R).

		Similarity Scores			LLM-as-a-Judge
Type	Model	$\bm{R_{L}}$ $\bm{\uparrow}$	BS $\bm{\uparrow}$	URL-R $\bm{\uparrow}$	FE $\bm{\uparrow}$	CA $\bm{\uparrow}$	FA $\bm{\uparrow}$	C $\bm{\uparrow}$	H $\bm{\uparrow}$	Domain Credibility	Domain Bias	# URL Used	Char Count	Coverage (# of Notes)
	Human-written Note	—	—	—	—	—	—	—	—	0.96	-0.377	1.668	172	488
	Llama-3.1-8B	0.193	0.864	0.12	4.087	3.585	2.479	2.464	2.796	0.935	-0.55	1.808	225	265
	Ministral-8B	0.208	0.861	0.125	4.083	3.589	2.487	2.355	2.672	0.954	-0.445	1.8	154	265
	Qwen3-14B	0.202	0.868	0.122	3.974	3.728	2.615	2.487	2.838	0.956	-0.396	1.525	147	265
	Apriel-Nemotron-15B	0.192	0.862	0.107	3.879	3.709	2.540	2.528	2.811	0.954	-0.329	1.664	227	265

Appendix

Table 7: Similarity scores, notes-level statistics, and LLM-as-a-Judge results for AI-generated CNs, using human-written helpful notes as ground truth. Metrics: ROUGE-L (

R_{L}

), BERTScore (BS), and URL Recall (URL-R). LLM-Judge (1–5): Functional Errors (FE), Claim Alignment (CA), Fact Alignment (FA), Completeness (C), and Helpfulness (H). Source Quality: Domain Credibility (0–1) and Domain Bias (-3 to +3). Gray-shaded rows report differences (

\Delta

); green / red coloring is shown only for GitSearch comparisons. Statistical significance (using independent t-test): *

p

<

0.05

, **

p

<

0.01

, ***

p

<

0.001

Term	Coef.	SE	$p$
Intercept	3.410	0.139	$<$ 0.001
GitSearch	0.520	0.190	0.007
SN Lite	$-$ 0.228	0.190	0.232
Web Agent	0.109	0.192	0.569
Length (chars)	$-$ 0.0003	0.000	0.172

Appendix

Table 8: OLS regression of human helpfulness (

H

) on note length and system identity (reference: Human).

N=400

R^{2}=0.042

Term	Coef.	SE	$p$
Intercept	$-$ 0.260	0.117	0.026
GitSearch	1.093	0.171	$<$ 0.001
SN Lite	$-$ 0.399	0.168	0.018
Web Agent	0.354	0.165	0.032
Length difference	$-$ 0.0001	0.000	0.296

Appendix

Table 9: Logistic regression of pairwise preference (row beats column) on length difference and system identity of the row method (reference: Human).

N=1{,}200

ordered comparisons, pseudo-

R^{2}=0.051

Fact-checked Tweet	Rank	Method	H	Note Content (Excerpt)
“Absolutely insane that Democrats actually voted to have the US government publish chld prn. WTF is even going on anymore???”	1	GitSearch (Gemini-2.5-Flash)	5	Claims that Democrats voted to publish chld prngrphy are false. The Epstein Files Transparency Act explicitly prohibits releasing chld sx**l abuse material and victims’ personal information.
	2	Human-written CN	4	The Ro Khanna amendment does not compel the government to publish chld prngrphy; the sponsor clarified that the DOJ would protect victims’ identities.
	3	Supernotes-Lite (Gemini-2.5-Flash)	3	Publishing chld prngrphy is illegal under U.S. law; proposed amendments aim to ensure compliance with existing prohibitions.
	4	Web-Search Agent (Gemini-2.5-Flash)	2	U.S. law criminalizes the production and distribution of chld prngrphy, but does not address the specific legislative claim.
“FIRED.”	1	GitSearch (Gemini-2.5-Flash)	4	Jerome Powell was not fired on July 16, 2025; President Trump stated that firing him was “highly unlikely,” resolving the ambiguity of the original post.
	2	Supernotes-Lite (Gemini-2.5-Flash)	3	Jerome Powell has not been fired; the post reflects a preference rather than a factual claim.

Appendix

Table 10: Error-analysis examples, with fact-checked tweets shown in the first column. Rows are ordered by human preference (Rank); our method ( GitSearch ) is in bold . We have not included the URLs’ text in the notes for this tabular presentation, and to mask sensitive keywords, we use asterisks (*).

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 139 papers

How Human Feedback Shapes AI-generated Community Notes
Jun 29, 2026Soham De, Isaac Slaughter, Jiawei Guo +4Claim Verification Automated Fact-Checking
University of Washington
Community-Driven API and AI Writer Design for Openly Scaling Community Notes
Sep 30, 2026Brad Miller, Jay Baxter, Jiansong Chao +3Human-in-the-Loop AI Automated Fact-Checking
Community Notes, SpaceXAI
iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics
Mar 4, 2026Preetam Prabhu Srikar Dammu, Arnav Palkhiwala, Tanya Roosta +1AI Agent Benchmarks LLM Agent Evaluation
University of Washington Seattle, Washington, USA · UC Berkeley Berkeley, California, USA

Jun 29, 2026J/K move · Enter open · S save

How Human Feedback Shapes AI-generated Community Notes

Soham De, Isaac Slaughter, Jiawei Guo, Qiao-Yun Cheng+3

University of Washington

Claim Verification Automated Fact-Checking Human-AI Collaboration Human-AI Co-Creation

Open paper AskPDF arXiv

Community Notes, a bridging-based crowd-sourced fact-checking system, has emerged as a new mechanism for moderating misleading information on social media and has been adopted by major platforms including X, Facebook, Instagram, Threads, and TikTok. Since its introduction, there has been an open question about what role AI could play in scaling and optimizing the system. Recently, X extended its Community Notes system by introducing Collaborative Notes: notes initially drafted by an LLM and iteratively refined based on feedback from human contributors. In this work, we systematically analyze the complete corpus of 19,146 collaborative notes and 211,850 instances of human feedback. First, we develop a taxonomy of human suggestions for improving AI-generated note drafts and find that suggestions involving factual corrections and additional context are most likely to be incorporated, while subjective policy judgments rarely are. Second, we examine changes in helpfulness across versions of collaborative notes and find that human feedback leads to more helpful notes, with the greatest impact coming from suggestions that challenge the main claim in the previous draft, particularly when submitted by more active contributors. Finally, we find that although collaborative notes improve through human feedback, they reach helpful status and are shown on the platform at lower rates than human-only or AI-only notes, with limited human participation emerging as a key bottleneck. Nevertheless, rather than serving as a weaker substitute, collaborative notes tend to play a complementary role, predominantly targeting posts that do not attract human-only or AI-only notes. Our analysis provides an initial description of efforts to use AI to improve crowdsourced content moderation in a real-world moderation system and outlines pathways for future improvements to such features.

Abstract

Automated Fact-Checking

Explore similar work

How Human Feedback Shapes AI-generated Community Notes

Community-Driven API and AI Writer Design for Openly Scaling Community Notes

iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics