ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models
Organizations: University of Pennsylvania, Philadelphia, PA, USA
Abstract
We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.
Figures & tables
| Model | LLMAggreFact | SciFact |
|---|---|---|
| BAcc | BAcc | |
| GPT-4 (zero-shot) | 75.3 | – |
| GPT-4o (zero-shot) | 75.9 | – |
| Claude-Sonnet-3.5 (zero-shot) | 77.2 | – |
| AlignScore (355M/fp16) | 70.4 | – |
| MiniCheck (7B/fp16) | 77.4 | 50.0 |
| Model | LLMAggreFact | SciFact | GSMClaims |
|---|---|---|---|
| BAcc | BAcc | Acc | |
| MiniCheck-7B | 77.4 | 50.0 | 51.3 |
| ThinknCheck-nothink-1B | 57.5 | 21.7 | 49.4 |
| ThinknCheck-1B | 78.1 | 64.7 | 52.2 |
| ThinknCheck-Science-1B | 79.2 | 66.4 | 61.0 |