cs.CLOct 25, 2026
SaveHigher Reasoning Effort Improves Reference Verifiability and Changes Reasoning Summaries in GPT-5.5
Organizations: Vrije Universiteit Brussel, Brussels, Belgium · Harvard University (SEAS), Cambridge, MA, United States
Abstract
Large language models are increasingly being used in different stages of scientific knowledge production, including the suggestion of references. These models can operate at different levels of reasoning effort and provide summaries of their reasoning processes. This raises two questions: Does increasing reasoning effort improve the verifiability of generated references, and do the resulting reasoning summaries reveal how the model's behavior changes with greater effort? We examined summaries from GPT-5.5 as it generated scholarly references for 1,000 papers under paired low- and high-effort settings. Generated references were matched to external bibliographic records, and reasoning summaries were scored for four predefined features and explored for additional features discovered by GPT-5.5. High effort increased the mean paper-level resolver-verified reference rate from 78.3% to 84.8%, and produced longer reasoning summaries. All four predefined features received higher scores under high effort. The discovered features changed less uniformly: some appeared equally often in both conditions, whereas others were more common under high effort. Within each effort condition, neither feature set showed a consistent increase in verification rates with higher scores.
Figures & tables
Figure 1: Study design and analysis pipeline. GPT-5.5 generates references and a reasoning summary for each of 1,000 focal papers under low and high reasoning effort. Top: Generated references are parsed, their metadata extracted using GPT-4.1 mini, and they are matched to external bibliographic records to determine verification status and metadata agreement. The resulting paper-level verified-reference rates are compared between low- and high-effort settings, overall and by research field. Bottom: Reasoning summaries are scored on predefined features and on model-discovered features derived by GPT-5.5, using a 1–5 scale from absent to strong, with GPT-5.5 and Claude Opus 4.8 serving as judges. (Figures 3 – 4 ).
Figure 2: Effect of reasoning effort on summary length and verified-reference rates. Paired observations for 1,000 focal papers. (A) , Distribution of reasoning-summary length in words. Curves are smoothed density estimates of summary lengths, each scaled to the tallest histogram bar of its condition. (B) , Number of focal papers in each research field; these are the denominators for panels (C) and (E). (C) , Summary length by field. Horizontal bars show the mean 1 between-paper standard deviation. (D) , Distribution of paper-level verified-reference rates under low and high reasoning effort. The purple distribution shows, for each paper, the percentage of its low-effort references that also appear in the high-effort list and are verified. Upper curves are count-scaled density estimates (dashed for shared). Each point below represents one focal paper, with hollow markers for shared rates. Thick black bars show the IQR. (E) , Mean verified-reference rate by field. (pp) gives the high-minus-low difference in percentage points.
Figure 3: Predefined feature expression in reasoning summaries. Columns show four predefined features, each scored 1–5 by a GPT-5.5 judge. A feature counts as present when its score is 3 or higher. (A) , Percentage of papers at each score, comparing effort conditions. (B) , Paper-level verified-reference rates grouped by feature score within each condition. Boxes show medians and IQRs; lines connect group means. (C) , Percentage of paper-condition observations in which the feature is present, in 50-token summary-length bins. Error bars show Wilson 95% confidence intervals ( Wilson, 1927 ) . Missing summaries are plotted at 0 tokens; their zero feature prevalence follows from this coding. Lengths above 600 tokens are pooled, and bins with fewer than five observations are omitted.
Figure 4: Model-discovered features in reasoning summaries. (A) , Papers expressing each feature, stacked by score levels of 3 or higher. (B) , Number of papers in which each feature is present in each condition. (C) , Mean percentage of generated references verified as real, by feature score and effort condition. The vertical scale spans 70–100%. (D) , Features present per paper by summary length (top: low effort; bottom: high effort). Box width reflects bin size; boxes show the IQR and white lines the median.
| Stage | Model | Settings |
|---|---|---|
| Citation generation | gpt-5.5-2026-04-23 | Responses API via Batch API (24-h window); reasoning effort low or high; reasoning summary auto (returned as detailed ); no output-token limit or sampling parameters set. |
| Metadata extraction | gpt-4.1-mini-2025-04-14 | Chat Completions API via Batch API (24-h window); temperature 0. |
| Predefined-feature scoring | gpt-5.5-2026-04-23; claude-opus-4-8 | GPT-5.5: Responses API via Batch API; reasoning effort medium; strict JSON-schema output. Claude Opus 4.8: Message Batches API; forced tool call returning the four scores; maximum 512 output tokens; extended thinking not enabled. |
| Feature discovery | gpt-5.5-2026-04-23 | Responses API via Batch API; reasoning effort medium; strict JSON-schema output; one request per condition and 200-paper batch. |
| Model-discovered feature scoring | gpt-5.5-2026-04-23; claude-opus-4-8 | GPT-5.5: Responses API via Batch API; reasoning effort medium; strict JSON-schema output with one integer 1–5 per feature. Claude Opus 4.8: Message Batches API; forced tool call with one integer 1–5 per feature; maximum output tokens ; extended thinking not enabled. |
Table S1: Model identifiers and request settings.
Figure S1 : Publication years of generated references. (A) The upper plot shows the publication-year distribution of the 1,000 focal papers (annual percentages and a smoothed curve). The lower plot shows density contours for generated-reference year against focal-paper year under low (blue) and high (red) reasoning effort. The dashed diagonal marks equal years; the shaded area above it contains references dated after their focal paper. (B) Distribution of the difference between reference year and focal-paper year for generated references with available years (low effort, ; high effort, ) and references recorded for the same focal-paper. Bars show annual relative frequencies; curves are smoothed visual guides. Differences below and above years are included in the endpoint bins. Positive differences indicate references dated after the focal paper. The observed references provide a descriptive comparison, not a criterion for verifying generated references.
Figure S2 : Candidate-match scores and resolver outcomes under high reasoning effort. Generated references are plotted by title- and author-match scores against a retrieved candidate record. Colors indicate the final resolver classifications: verified, possible, or unresolved. The dotted boundary marks the possible-match thresholds (title , author ); the solid boundary marks the verified-match thresholds (title , author ). Verification also required agreement of the normalized first-author surnames, so meeting the plotted score thresholds alone did not guarantee a verified classification. For possible matches, the qualifying candidate among the three highest-ranked candidates is shown when it differs from the candidate retained in the output record. The axes display scores from 0.2 to 1.0; class counts include references whose scores fall outside this displayed range.
| Lowest | Highest | |
| Low effort | ||
| Threshold (verified rate) | 55% | 95% |
| Papers | 124 | 194 |
| Mean verified-reference rate | 42.6% | 96.6% |
| Epistemic caution | 2.10 | 1.93 |
| Temporal-constraint awareness | 1.06 | 1.06 |
Table S2 : Mean predefined-feature scores for focal papers with the lowest and highest verified-reference rates. Within each reasoning-effort condition, the lowest group comprises papers at or below the 10th percentile of paper-level verified-reference rate and the highest group papers at or above the 90th percentile. Because the rate is discrete (steps of about 5 percentage points), all papers tied at a threshold are included, so group sizes differ from 100.
| Feature | Representative model-provided definition |
|---|---|
| Reference count | The summary explicitly plans to gather a specific approximate number of sources, most commonly around 20 citations or references. |
| Topical relevance | The summary emphasizes that each citation should directly support the focal topic or user need rather than merely being generally related. |
| Authenticity concern | The summary states that references should be real, actual, credible, trustworthy, or not fabricated/hallucinated. |
| Bibliographic accuracy | The summary expresses an intent to double-check, verify, or ensure correctness of bibliographic details and source reliability. |
| Citation formatting | The summary explicitly considers formatting requirements, citation style, numbered lists, APA, Spanish style, or otherwise presenting references consistently. |
| Relevance notes | The summary plans to add brief descriptions explaining why each reference is relevant to the topic. |
Table S3 : Definitions of the eleven model-defined reasoning features.
Figure S3 : Robustness of feature scores to the choice of judge. GPT-5.5 reasoning summaries were scored by GPT-5.5 (filled) and Claude Opus 4.8 (open) under low (blue) and high (red) reasoning effort. Each comparison uses observations with valid scores from both judges. (A) Mean scores for the four predefined features on the 1–5 ordinal scale, using 994 low-effort and 995 high-effort observations. (B) Percentage of observations in which each of the eleven harmonized model-discovered Features is present, defined as a score of at least 3, using 996 low-effort and 995 high-effort observations. Gray lines connect the two judges’ estimates within each effort condition.
| Feature | Effort | Score | Exact phrase from reasoning summary |
|---|---|---|---|
| Epistemic caution | Low | 3 | “I need to provide relevant references that are likely real for the topic of chitosan adhesive and its connection to mussels, dopamine, and tyrosinase.” |
| High | 4 | “I need to verify the difference between ‘bone morphogenic’ and ‘morphogenetic.’ It seems like the title might refer to ‘bone morphogenic protein antagonist gremlin…’ ” | |
| Temporal constraint awareness | Low | 4 | “It’s crucial to ensure that these include prior studies published before 2012.” |
| High | 4 | “I’m wondering if Simonoff’s 2008 work would be appropriate to include after the target publication date.” | |
| High | 5 | “However, since the paper was published in 2000, I realize that citations post-2000 can’t be used in the original context.” | |
| Reference verifiability checking | Low | 3 | “I need to provide relevant references that are likely real for the topic of chitosan adhesive and its connection to mussels, dopamine, and tyrosinase.” |
Table S4 : Illustrative examples of predefined reasoning-feature scores under low and high reasoning effort.
More in this topic
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 802 papers