Mathematical Reasoning Enhanced LLM for Formula Derivation: A Case Study on Fiber NLI Modellin

Within the past few years, the ability of Large Language Models (LLMs) to generate formal mathematical proofs has improved drastically. We provide a comparison of various LLMs' effectiveness in producing formal proofs in Lean 4 with the goal of assisting those seeking to use LLMs to support their own projects. We utilize both pass@$k$ and refine@$k$ metrics as the benchmark for our comparison and evaluate on subsets of both miniF2F and miniCTX datasets. Our testing shows that overall, Gemini 3.1 Pro and Claude Opus 4.7 perform best. Gemini 3.1 Pro achieved a 92% success rate on miniF2F via refine@32 whereas Opus 4.7 achieved a 86% success rate on miniCTX via refine@32. When taking cost into account, NVIDIA Nemotron 3 Super and GPT-OSS 120B were the most efficient, with competitive accuracies and average costs of <\0.01$ per correct proof.

Large Language Models (LLMs) have shown strong capabilities across natural language tasks. But their ability to generate accurate written verbalizations of mathematical formulas has not been systematically evaluated. Such verbalization is important for text-to-speech systems supporting audio-based learning and visually impaired users. We evaluated five state-of-the-art LLMs across six scientific domains and constructed Read-Aloud Math, a balanced multi-domain corpus with LaTeX formulas, manually validated simple and extended transcriptions, multiple variants, textual context, domain labels, and formula complexity levels. Using multi-reference BLEU, GPT-5.2 achieved the highest score for simple transcriptions (75.9% without context, 64.9% with context), although without context it did not significantly outperform Grok 4.1 Fast or Claude Sonnet 4.5. Claude Sonnet 4.5 achieved the highest score for extended transcriptions without context (36.5%), and Grok 4.1 Fast with context (48.8%). For extended transcriptions, Statistics and Mathematics scored highest across the two context settings, while Physics scored lowest. Formula length was significantly negatively associated with BLEU for simple transcriptions without context, but not with context, whereas BLEU increased significantly with formula length for extended transcriptions in both settings. Grouping symbols were most frequently involved in transcription differences, while variable names, capitalization, and Greek letters were associated with lower BLEU scores. Overall, LLM-based formula verbalization is promising but remains sensitive to transcription type, textual context, and formula characteristics.

This work investigates the ability of large language models (LLMs) to generate mathematical equations from scientific texts. Prior work faces challenges in unstructured grounding, multi-equation dependency, and humanaligned evaluation. To this end, we construct a dataset of AI research papers, pairing contextual passages with ground-truth equations and variable descriptions. We develop an explainable equation generation workflow and evaluate it across diverse open- and closed-source LLM backbones. We introduce an evaluation protocol combining automatic metrics, LLM-based rubrics, and human judgments to assess accuracy, explainability, and human-LLM alignment. Results indicate that LLMs perform moderately on lexical- and syntactic-based similarity, while struggling with semantic accuracy. Comparisons between LLM-based evaluations and human judgments reveal limited alignment, highlighting challenges in using LLMs to assess equation quality. These findings offer insights for improving equation generation models and developing more reliable evaluation methods for scientific text. We provide code and data for reproducibility.