cs.LGOct 12, 2026

ACED: Reference-Anchored Aggregation of Natural-Language and Executable Evidence for Mathematical Reasoning

Authors: Yiyi Lu, Jiaxiang Geng, Bing Luo

Organizations: 1Duke Kunshan University · 2The University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Natural-language self-consistency and executable programs provide complementary test-time evidence for mathematical reasoning, yet either channel may support the same semantic error. Thus we propose ACED, a reference-anchored framework that treats the self-consistency answer as the default and executable outputs as supporting evidence rather than interchangeable votes. ACED converts source-specific answer counts into smoothed supports, combines them through a weighted logarithmic pool, and overrides the reference only when a calibrated relative-evidence margin is exceeded. The same rule supports both candidate selection and admission of executable-only answers. Across five math word-problem benchmarks, ACED matches or outperforms the corresponding baselines. On GSM8K, ACED-CANDIDATE recovers 89.7% of the attainable candidate-set gain with only three damaging overrides, while ACED-ALL reaches 86.9% accuracy versus 82.6% for direct Program-of-Thought aggregation. These results show that preserving evidence provenance enables more reliable integration of natural-language and executable reasoning.

Figures & tables

Figure 1: Overview of ACED. (a) Self-consistency offers the reference answer and problem-only programs provide a separate evidence source. (b) The selector operates over candidate-only and all-feasible answer domains. (c) Outcomes distinguish retained correct references, successful fixes, damaging replacements, and unresolved errors.
AnswerNL supportProgram supportPooled count
AA909
BB178
C,D,EC,D,E (each)011
Table 1: A minimal conflict in which pooled voting favors AA , while add-one product support favors BB .
(a) Five benchmarks NL reference: Qwen2.5-0.5B-Instruct
 Accuracy (%) ↑\uparrowGSM8K overrides
MethodGSM8KSVAMPASDivMultiArithMAWPSFix ↑\uparrowDamage ↓\downarrow
SC@16 (reference)31.854.059.097.884.1––
Candidate-only selection
ACR31.150.758.696.784.17281
ProVe49.668.071.1100.090.72449
Table 2: Results across five benchmarks with a 0.5B natural-language reference in (a) and on GSM8K with a stronger 7B reference in (b). Fix and Damage are measured relative to the corresponding SC reference, and pp -values in (b) are obtained from exact paired McNemar tests against Strong-SC. Shaded rows denote ACED. Bold indicates the best accuracy, correct count, Fix, and Damage within each answer domain in (a) and across selectors in (b), excluding reference and oracle rows.
Figure 2: GSM8K selection and inference cost. (a) Oracle-gap utilization for candidate selection. (b) Witness construction and selector contributions at M=16M=16 , with the last two configurations sharing witnesses. (c) Correct predictions versus wall time, with program budgets labeled and arrows marking the reported cost comparisons. Filled markers and horizontal bars show mean time and one standard deviation over three runs, while open markers show single measurements. Correct counts come from the reported evaluation, not averages over timing runs.
MethodCorrect ↑\uparrowTokens / question ↓\downarrowTime / question (s) ↓\downarrowSpeedup ↑\uparrow
 ExactWeightedGenerationExecutionWall 
Lower-cost comparison
PoT@3210987631.253418.13.93 ±\pm 0.350.44 ±\pm 0.014.37 ±\pm 0.341.00 ×\times
ACED-ALL@811226187.921241.02.51 ±\pm 0.220.12 ±\pm 0.012.63 ±\pm 0.211.66 ×\times
Higher-accuracy comparison
PoT@64113115931.1111517.47.34 ±\pm 0.710.88 ±\pm 0.028.21 ±\pm 0.691.00 ×\times
Table 3: GSM8K accuracy and inference cost. Token counts include input and output, with model-size weights of 0.5 for natural-language tokens and 7 for code tokens in the weighted column. Timing is mean ±\pm std over seeds 13, 21, and 42 on the same setup. Speedup divides the mean PoT wall time by the method’s mean wall time within each block.
ConfigurationCorrectFixDamage
Direct PoT pool + executable majority109069221
ACED witness pool + executable majority11437339
ACED witness pool + ACED selector11467336
Table 4: GSM8K source–selector decomposition at M=16M=16 . The first two rows hold executable-majority selection fixed while comparing separately generated problem-only witness pools. The last two rows reuse exactly the same ACED witnesses and therefore isolate the effect of the ACED selector.
Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Input: problem xx ; budgets N,MN,M ; domain Ω∈{Ωcand,Ωall}\Omega\in\{\Omega_{\mathrm{cand}},\Omega_{\mathrm{all}}\} ; smoothing βN,βP>0\beta_{N},\beta_{P}>0 ; frozen λ,τ≥0\lambda,\tau\geq 0 .
Output: selected answer a^Ω(x)\hat{a}_{\Omega}(x) .
Appendix
Algorithm 1 Information pooling and calibrated acceptance
ItemValue
Calibration data500 GSM8K training examples, comprising 350 conflict and 150 non-conflict cases, selected with seed 13 and disjoint from the test sets.
Evidence budgetsN=16N=16 , with M∈{8,16,32}M\in\{8,16,32\} for the reported settings.
Evidence weightsλ∈{0.25,0.5,1,2,4,8}\lambda\in\{0.25,0.5,1,2,4,8\}
Thresholdsτ∈{0,0.25,0.5,1,1.5,2,3,4}\tau\in\{0,0.25,0.5,1,1.5,2,3,4\}
Selection criterionHighest correct count among settings with at most 10 damaging overrides, or 2%2\% of the full split.
Tie resolutionFixed representatives in Table 6 , chosen before test evaluation.
Appendix
Table 5: Calibration data, search grid, and operating-point selection.
   Calibration ( n=500n=500 )GSM8K test ( n=1319n=1319 )
Settingλ\lambdaτ\tauCorrectFixDamageOverridesCorrectFixDamageOverrides
Cand@164029013701587693533421
Cand@322029113801617733573437
All@841473324435011227107841
All@1640472323435011467336864
All@3241473324435011517386867
Appendix
Table 6: Frozen operating points and their calibration and test outcomes. Cand and All denote ACED-CANDIDATE and ACED-ALL, and the suffix gives MM . The calibration and test SC references are correct on 153 and 419 examples, respectively. Overrides include wrong-to-wrong changes as well as Fix and Damage.
Figure 3: Calibration and GSM8K override behavior at M=16M=16 . (a) Best calibration Fix count under each Damage budget, with stars at the frozen settings. (b) Fix rate among the 900 SC-wrong test examples. (c) Damage rate among the 419 SC-correct test examples.
Calibration setting0% cap1–5% caps
Cand@16290/137/0290/137/0
Cand@32291/138/0291/138/0
All@8327/174/0473/324/4
All@16341/188/0472/323/4
All@32288/135/0473/324/4
Appendix
Table 7: Calibration-cap sensitivity on the fixed 500-example split. Each entry is Correct/Fix/Damage for the best eligible point. Caps of 1%, 2%, and 5% select the same plateau.
Domain / MMCorrectFixDamage
Cand / 16769–769353–3533–3
Cand / 32771–773356–3573–4
All / 81109–1126697–7147–7
All / 161134–1146721–7336–6
All / 321144–1151731–7386–6
Appendix
Table 8: Sensitivity to βN,βP∈{0.5,1,2}\beta_{N},\beta_{P}\in\{0.5,1,2\} . Each setting recalibrates λ\lambda and τ\tau under the same pre-test rule. Entries give the range over the nine smoothing pairs. The unit-smoothing operating points appear in Table 6 .
 τ\tau
λ\lambda00.250.511.5234
ACED-CANDIDATE@ M=16M=16
0.25269/116/0234/81/0188/35/0153/0/0153/0/0153/0/0153/0/0153/0/0
0.5290/137/0290/137/0290/137/0223/70/0153/0/0153/0/0153/0/0153/0/0
1290/137/0290/137/0290/137/0290/137/0290/137/0274/121/0153/0/0153/0/0
2290/137/0290/137/0290/137/0290/137/0290/137/0290/137/0290/137/0290/137/0
Appendix
Table 9: Complete calibration sweep for the five reported domain and budget settings. Each cell is Correct/Fix/Damage on 500 GSM8K training examples. Bold marks the frozen representative selected from the highest-correct eligible plateau.
MethodGSM8KSVAMPASDivMultiArithMAWPS
Candidate-only selection
SC@16419/131981/150151/25688/90808/961
ACR410/131976/150150/25687/90808/961
ProVe654/1319102/150182/25690/90872/961
ACED-CANDIDATE@16769/1319119/150207/25690/90896/961
All-feasible selection
Appendix
Table 10: Exact correct counts corresponding to the five-benchmark main table, with the additional ACED-ALL@32 row shown for budget scaling. Bold marks the best M=16M=16 result within each answer-domain block, including ties.
ComparisonA-onlyB-onlyΔ\DeltaExact pp
ACED-ALL@8 vs. PoT@810158+438.11⋅10−48.11{\cdot}10^{-4}
ACED-ALL@16 vs. PoT@1610549+567.50⋅10−67.50{\cdot}10^{-6}
ACED-ALL@32 vs. PoT@3210350+532.20⋅10−52.20{\cdot}10^{-5}
Appendix
Table 11: Matched-budget GSM8K comparisons. A-only and B-only count examples solved exclusively by the first and second methods. Δ\Delta is their difference, and pp is the exact paired McNemar result.
Figure 4: Results across code budgets, benchmarks, and generation seeds. (a) GSM8K correct counts as the code budget increases. (b) Accuracy on the four additional benchmarks at M=16M=16 . (c) GSM8K correct counts over generation seeds 3, 13, and 21.
DatasetMethodCorrectFixDamage
SVAMPPoT@16136/150583
 ACED-ALL@16137/150604
ASDivPoT@16217/256715
 ACED-ALL@16222/256732
MAWPSPoT@16916/96112416
 ACED-ALL@16926/9611268
Appendix
Table 12: Correct counts, Fix, and Damage at M=16M=16 , relative to each dataset’s SC@16 reference. Bold marks the better value within each dataset and metric.
MMDecision ruleCorrectFixDamage
8Executable majority112471611
8ACED selector11227107
16Executable majority11437339
16ACED selector11467336
32Executable majority114773810
32ACED selector11517386
Appendix
Table 13: All-feasible decisions on the same ACED witness pool. Executable majority uses only vACED(a)v_{\mathrm{ACED}}(a) , while the ACED selector combines natural-language and executable support using the frozen operating point. Bold marks the better value within each budget and metric.
ComparisonA-onlyB-onlyMcNemar pp
ACED-CANDIDATE@16 vs. Strong-SC@16117185.72⋅10−195.72{\cdot}10^{-19}
ACED-ALL@16 vs. Strong-SC@16152363.67⋅10−183.67{\cdot}10^{-18}
ACED-ALL@16 vs. ACED-CANDIDATE@1639220.0396
PoT@16 vs. Strong-SC@16129920.0153
PoT@32 vs. Strong-SC@16136910.00341
Appendix
Table 14: Paired GSM8K counts with the stronger SC reference. A-only and B-only are examples solved only by the first or second method, respectively.
Method31321Mean ±\pm std
PoT@81076107910781077.7 ±\pm 1.5
PoT@161088109010891089.0 ±\pm 1.0
PoT@321096109810971097.0 ±\pm 1.0
ACED-CANDIDATE@16757769765763.7 ±\pm 6.1
ACED-ALL@81119112211241121.7 ±\pm 2.5
ACED-ALL@161141114611481145.0 ±\pm 3.6
Appendix
Table 15: Full-GSM8K correct counts for generation seeds 3, 13, and 21. Mean ±\pm std is computed across the three seeds.
MethodCorrectTokens/qW-gen/qW-total/qGen s/qExec s/qWall s/q
Direct executable aggregation
PoT@3210987631.216837.153418.13.93 ±\pm 0.350.44 ±\pm 0.014.37 ±\pm 0.34
PoT@64113115931.138355.5111517.47.34 ±\pm 0.710.88 ±\pm 0.028.21 ±\pm 0.69
Natural-language and executable evidence
ACED-ALL@811226187.910245.321241.02.51 ±\pm 0.220.12 ±\pm 0.012.63 ±\pm 0.21
ACED-ALL@1611468979.119686.240779.13.42 ±\pm 0.310.22 ±\pm 0.003.64 ±\pm 0.31
Appendix
Table 16: GSM8K inference cost per question. Tokens/q includes input and output. W-gen/q and W-total/q apply model-size weights of 0.5 and 7 to natural-language and code tokens, respectively. W-gen/q counts output tokens only. Timing is mean ±\pm std over seeds 13, 21, and 42, except for the single ACED-ALL@32 accounting run. Correct counts are reported evaluation results, not means over timing runs.
SettingValue
NL candidate modelQwen/Qwen2.5-0.5B-Instruct
Stronger SC reference modelQwen/Qwen2.5-7B-Instruct
Code witness / direct PoT modelQwen/Qwen2.5-Coder-7B-Instruct
Reported sample countsN=16N=16 and M∈{8,16,32}M\in\{8,16,32\}
Generation limits256 maximum new tokens for natural-language candidates and 192 for ACED witnesses and direct PoT programs
Code prompt limit1,536 tokens for code-generation prompts
Appendix
Table 18: Models, generation limits, and execution settings. Baseline-specific settings appear in Appendix F.3 .
ComponentImplementation entry point
NL prompt and extractionsrc/phase_a_tango.py : render_prompt , extract_gsm8k_answer , normalize_numeric_answer
Direct PoT promptsrc/exact_acv/exact_acv.py : build_solver_prompt , and src/race_acv/prompt_program_baseline.py : build_program_prompt
ACED witness promptsrc/race_acv/prompt_program_baseline.py : build_program_prompt , style repair_problem
ProVe promptsrc/race_acv/prove_baseline.py : build_prove_prompt
ACR promptsrc/race_acv/acr_option_baseline.py : build_prompt
Program extraction, filtering, executionsrc/exact_acv/exact_acv.py : extract_code , is_suspicious_program , execute_code
Appendix
Table 19: Source-level implementation hooks used for the reported experiments. Paths are relative to the implementation root.
InterfaceTemplate text
(a) Problem-only interfaces
NL candidate generatorSolve the following math problem. Reason step by step and put the final answer at the end. Problem: {question}.
PoTYou are given a grade-school math word problem. Write Python code that defines a function solve() . Rules: compute the answer from the quantities in the problem; do not hard-code the final answer; use only arithmetic and optional math, fractions, or decimal; return a single number from solve() ; output only Python code. Problem: {question}. Python code:
ACED witnessSolve this math word problem independently. Write only Python code. The code must define solve() and return one numeric answer. Before coding, audit the wording of the final question and identify the exact requested quantity. Then define variables and equations from the problem and compute that requested quantity. Do not hard-code the final answer. Do not return a subtotal, rate, leftover, or intermediate value unless it is exactly what is asked. Problem: {question}. Python code:
(b) Candidate-conditioned interfaces
ProVeYou are verifying a grade-school math solution by translating its arithmetic into Python. Write only Python code. The code must define solve() and return one numeric answer. Follow the quantitative steps in the provided solution. Do not simply return a final literal answer; derive it from the quantities and operations. Problem: {question}. Candidate solution: {candidate_solution}. Python code:
Appendix
Table 20: Prompt templates used by the five interfaces. Braced placeholders indicate inserted fields. The ellipsis in the ACR template denotes further answer options in the same format.
ItemSetting
Baseline typeAnswer-option rescoring baseline
PublicationDOI: 10.3390/app15179587
Candidate domainCandidate-only, stored N=16N=16 natural-language answer clusters
Teacher modelQwen/Qwen2.5-Coder-7B-Instruct code-model teacher
Options / votesTop K=8K=8 answer options and 5 teacher votes per question
Weight and ruleWeight 0.25. Select by nNL/16+0.25⋅vteacher/5n_{\mathrm{NL}}/16+0.25\cdot v_{\mathrm{teacher}}/5 .
Appendix
Table 21: ACR configuration under the paper’s full-GSM8K candidate reporting interface.
ArgumentValue
Repositoryhttps://github.com/declare-lab/prove
Commita5a8f3feabf05b340df4d9cf d72b2bf64bec6345
Entry / pipelineeval.py , pipeline prove
Dataset / samplesgsm8k , num_cot=16
CoT generationcot template, qwen2_05b , temperature 0.7, max tokens 1024
Answer extractionextract template, qwen25coder_7b , temperature 0.0, max tokens 32
Appendix
Table 22: ProVe repository configuration used for the candidate-domain baseline.
Output categoryNLACEDPoT
Outputs total21,10442,20842,208
Excluded before MvalidM_{\mathrm{valid}}05,3942,478
Retained rows ( NN or MvalidM_{\mathrm{valid}} )21,10436,81439,730
Numeric answer rows21,10434,46139,673
Fallback non-numeric keys000
No-answer null-bucket rows02,35357
Appendix
Table 23: Full-GSM8K output and normalization counts. ACED and PoT use M=32M=32 . Collision keys receive multiple raw forms that normalize to the same numeric answer.
Sandbox componentAllowed interface
Importsmath , fractions , and decimal
Builtinsabs , min , max , sum , round , range , len , float , int , str , pow , and print
Primary entry pointCallable solve() returning the answer
Fallback result namesresult , answer , ans , total , total_cost , final , and final_answer
Execution limit2.0 seconds per program
Appendix
Table 24: Restricted execution interface for generated Python programs.
ItemConfiguration
GPU infrastructure2×2\times NVIDIA A100
GPU memory80 GB per GPU
GPUs per timing job1
Numerical precisionbf16
Main NL modelQwen2.5-0.5B-Instruct (nominally 0.5B parameters)
Stronger NL modelQwen2.5-7B-Instruct (nominally 7B parameters)
Appendix
Table 25: Hardware and model configuration. GPU memory denotes device capacity, not measured peak allocation.
PackageVersion
PyTorch2.7.1
Transformers4.57.1
Accelerate1.13.0
Datasets4.8.5
NumPy2.2.6
Pandas2.3.3
Appendix
Table 26: Package versions specified for reproduction.

Explore similar work

CardsList

  1. Re2^2Math: Benchmarking Theorem Retrieval in Research-Level Mathematics

    May 9, 2026Zicheng Lyu, Wenjie Yang, Shengzhong Zhang +1Mathematical Reasoning BenchmarksLLM Evaluation

    Fudan University Shanghai, China · Nanjing University of Aeronautics and Astronautics Nanjing, Jiangsu, China · Fudan University Shanghai Innovation Institute Shanghai, China

  2. KACE: Knowledge-Adaptive Context Engineering for Mathematical Reasoning

    May 30, 2026Jayant Parashar, Suchendra M. BhandarkarMemory-Augmented Language ModelsSelf-Consistency Decoding

    School of Computing University of Georgia

  3. Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance

    Feb 26, 2026Weida Liang, Yiyou Sun, Shuyuan Nan +3LLM Mathematical ReasoningInference-Time Guidance

    National University of Singapore · University of California, Berkeley