$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient

Hongbo Ma; Sansheng Cao; Jiajun Fan; Bangji Yang; Ge Liu

cs.LGSep 29, 2026

$S^3$ : Spectral Null-Space Swap Makes Reasoning Models Efficient

Authors: Hongbo Ma, Sansheng Cao, Jiajun Fan, Bangji Yang, Ge Liu

Organizations: University of Illinois Urbana-Champaign · Tsinghua University · Dartmouth College · Peking University

Abstract

LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capacity lies in the Thinking model's weight component within the null space of a projection defined by the corresponding Non-thinking model's dominant singular directions, and removing the subspace component can largely improve reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose Spectral Null-Space Swap ( $S^3$ ), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving reasoning efficiency while maintaining accuracy. We extensively evaluate $S^3$ on 2B-30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across mathematical, multimodal, and audio reasoning domains. $S^3$ establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0% token speedup). We further use attention entropy for explanation and find that the retained component produces more concentrated attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce attention entropy, thereby improving the efficiency of reasoning.

Figures & tables

Figure 1 : This figure compares the parallel and perpendicular components of the thinking difference in parameter and function spaces. All panels use the Qwen3-4B-Instruct and Qwen3-4B-Thinking. Parameter space denotes weight differences; function space denotes hidden-state offsets relative to Qwen3-4B-Instruct on AIME24/25. (a) Global average results. (b) Per-layer parameter-space results. (c) Per-layer function-space results on the same inputs.

Figure 2 : Overview of our method: Spectral Null-Space Swap and the

\rho=1

probe family. The swapped weight preserves the protected spectral component of the Non-thinking model and imports the complementary component of the Thinking model. The four probes isolate the contributions of

\Delta W_{\parallel}

and

\Delta W_{\perp}

Large Language Model	AIME25		HMMT25		CMIMC25		Olympiad-Bench
Large Language Model	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$
Qwen3-4B
Qwen3-4B-Instruct	45.8	6,785	35.0	8,621	32.5	8,461	77.3	4,468
Qwen3-4B-Thinking	71.7	20,704	46.7	24,636	58.1	24,415	83.8	14,086
Qwen3-4B-TIES	73.3	16,377	50.0	16,049	54.4	21,748	85.8	10,238
Qwen3-4B-MI-0.8	71.7	17,270	58.3	19,729	56.3	22,902	85.5	11,460

Table 1 : Main results across text, vision–language, and audio reasoning benchmarks. We report accuracy (%) and average generated tokens per problem. Bold indicates the best value in each column among TIES, MI-0.8, and S 3 . In each

\Delta

row, accuracy is reported as an absolute change and token usage as a relative change with respect to the corresponding Thinking model.

Model	AIME25		HMMT25		CMIMC25		Olympiad-Bench
Qwen3-4B-S 3 -1.0	69.2	15,129	45.8	15,920	50.6	17,863	83.0	9,553
Qwen3-4B-S 3 -0.9	71.7	15,843	49.2	16,733	53.1	20,567	83.5	9,607
Qwen3-4B-S 3 -0.8	73.3	15,004	55.0	16,496	55.0	21,266	83.8	9,745
Qwen3-4B-S 3 -0.7	75.8	18,082	45.0	22,377	48.1	22,186	84.0	10,265
Qwen3-4B-S 3 -0.6	78.3	18,542	49.2	22,481	50.6	22,798	83.8	11,607
Qwen3-4B-S 3 -0.5	79.2	19,264	54.2	23,361	51.3	23,743	83.6	12,035

Table 2: Effect of the protected-subspace ratio in Qwen3-4B-S 3 on the four benchmarks reported in the main table. We report accuracy (%) and average generated tokens per problem. Bold indicates the best value in each column.

Model	AIME’24	AIME’25	AMC’23	CMIMC’25	HMMT’25	Mean Acc	Mean len
Base	61.7	50.4	94.7	30.6	33.3	54.1	8,280
Sub	69.2	56.2	95.9	36.2	35.4	58.6	8,891
Null	77.1	74.6	99.1	51.2	49.2	70.2	15,071
Full	75.4	70.0	99.1	53.4	53.3	70.3	20,269

Table 3: Accuracy and token length across five math benchmarks generated by the probe family. The Sub model is almost identical to the Base model. The Null model matches the Full model’s average accuracy while generating fewer tokens.

Figure 3 : Attention entropy analysis across four model variants. (a) Mean attention entropy averaged over layers 32-35: Null consistently achieves the lowest entropy while Sub exhibits the highest; additional results are shown in Appendix A.2 . (b) Top-8 attention weights at layers 32-35 show that Null concentrates attention. (c) Per-problem entropy scatter on AIME at layer 35: the ordering $H_{\text{Null}}<H_{\text{Base}}<H_{\text{Sub}}$ holds for all the individual problems.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4 : Layer 32–35 average attention entropy for the four variants on all seven datasets. The ordering

H_{\mathrm{Null}}<H_{\mathrm{Base}}<H_{\mathrm{Sub}}

holds throughout; Full lies between Null and Sub in all seven datasets (values in Table 4 ).

Model	AIME24	AIME25	AMC23	CMIMC25	HMMT25	MATH-500	GSM-8K	All
Null	0.871	0.981	0.836	0.882	0.894	0.792	0.815	0.861
Base	0.884	0.996	0.845	0.897	0.904	0.809	0.823	0.873
Full	0.887	1.006	0.855	0.907	0.916	0.803	0.824	0.878
Sub	0.896	1.011	0.856	0.912	0.917	0.820	0.834	0.886

Appendix

Table 4: Mean attention entropy (nats) averaged over layers 32-35

Model	Acc. (%)	Hall. rate (%)	Correct	Incorrect	Not attempted
Base	19.1	80.4	826	3384	116
Sub	15.8	81.3	682	2969	675
Null (ours)	20.2	78.7	873	3225	228
Full	19.6	80.4	847	3466	13

Appendix

Table 5: SimpleQA results for the Qwen3-30B-A3B variants. The ordering Null

<

Base

<

Sub mirrors the attention entropy ordering of Fig. 3 .

Figure 5 : Attention entropy versus pretraining step (Pythia-1.4B), measured on fixed chain-of-thought input. Both curves converge to their final levels (dashed lines).

Figure 6 : The null-space component negatively aligns with the entropy gradient across all benchmarks.

G_{\perp}

is the layer 32–35 mean attention entropy gradient at Base , and

\Delta W_{\perp}

is the null-space component. Left axis: inner product

\langle\Delta W_{\perp},G_{\perp}\rangle

across all weight matrices, which are consistently negative. Right axis: Angle between the two vectors.

Model family	Architecture	Modality	Task group
Qwen3-4B	Dense	Text	Math and general reasoning
Qwen3-30B-A3B	MoE	Text	Math and general reasoning
Qwen3-VL-2B	Vision–language	Text and image	Multimodal reasoning
Qwen3-VL-4B	Vision–language	Text and image	Multimodal reasoning
Qwen3-Omni-30B-A3B	MoE	Text and audio	Text and audio reasoning

Appendix

Table 6: Model families and evaluation coverage.

Setting	Temperature	Presence penalty	Max tokens	Max Model Length	Backend
Text reasoning	0.7	1.5	32,768	40,960	vLLM
VL–text	0.7	1.5	32,768	40,960	vLLM
VL–multimodal	0.7	0.0	16,384	40,960	vLLM
Omni–text	0.7	1.5	32,768	40,960	vLLM
Omni–audio	0.6	0.0	16,384	32,768	vLLM

Appendix

Table 7: Task-specific generation configurations.

Model	AIME24		AIME26		AMC23		MATH-500
Model	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$
Qwen3-4B
Qwen3-4B-Instruct	63.3	7,951	51.7	7,769	93.8	2,904	94.5	1,843
Qwen3-4B-Thinking	73.3	19,247	74.2	20,461	100.0	10,443	97.7	6,161
Qwen3-4B-TIES	70.8	14,905	76.7	15,029	100.0	8,720	98.2	5,152
Qwen3-4B-MI-0.8	83.3	15,262	80.0	16,165	99.4	8,059	98.1	4,879

Appendix

Table 8 : Additional LLM results on math reasoning benchmarks. We report accuracy (%) and average generated tokens per problem. Bold indicates the best value in each column among TIES, MI-0.8, and S 3 .

Figure 7 : Accuracy–token Pareto frontiers across eight LLM reasoning benchmarks. Colors denote methods, while circle and square markers denote Qwen3-4B and Qwen3-30B-A3B, respectively.

Figure 8 : Accuracy–token Pareto frontiers across four vision–language benchmarks. Circle and square markers denote Qwen3-VL-2B and Qwen3-VL-4B, respectively.

Figure 9 : Accuracy–token Pareto frontiers across four Omni benchmarks.

Qwen3-30B-A3B
Model	AIME25		HMMT25		CMIMC25		Olympiad-Bench
Qwen3-30B-A3B-S 3 -1.0	74.2	11,152	57.5	13,454	60.6	14,110	86.7	7,132
Qwen3-30B-A3B-S 3 -0.9	80.0	11,640	64.2	15,603	68.8	15,472	86.9	8,599
Qwen3-30B-A3B-S 3 -0.8	80.8	12,536	62.5	16,424	66.3	15,988	88.4	9,238
Qwen3-30B-A3B-S 3 -0.7	85.0	13,915	62.5	19,403	71.3	17,727	86.8	9,939
Qwen3-30B-A3B-S 3 -0.6	77.5	16,048	65.8	20,548	68.1	20,091	86.4	10,256

Appendix

Table 9: Effect of the protected-subspace ratio across Qwen3-30B-A3B and Qwen3-VL models. We report accuracy (%) and average generated tokens per problem. Bold indicates the best value in each column within each model family.

Problem: Let the sequence of rationals $x_{1},x_{2},\dots$ be defined such that $x_{1}=\frac{25}{11}$ and $x_{k+1}=\frac{1}{3}\left(x_{k}+\frac{1}{x_{k}}-1\right).$ $x_{2025}$ can be expressed as $\frac{m}{n}$ for relatively prime positive integers $m$ and $n$ . Find the remainder when $m+n$ is divided by $1000$ .
Ground Truth: 248
Qwen3-4B-Thinking (32,768 tokens) S 4 = m 3 2 + n 3 2 - m 3 n 3 $\Rightarrow$ S 4 mod 3000 = 1089 + 625 - 2775 = (1089 + 625) = 1714; 1714 - 2775 = -1061 $\Rightarrow$ -1061 + 3000 = 1939 $\Rightarrow$ S 4 $\equiv$ 1939 mod 3000 Now, 3m 4 $\equiv$ 1939 mod 3000. But 1939 must be divisible by 3? Wait, 1+9+3+9=22, not divisible by 3. Contradiction! But we know from earlier that m 4 is an integer, so S 4 is divisible by 3. Therefore, my mod 3000 calculation is incorrect because I used m 3 mod 1000=33, but m 3 =19033, so m 3 mod 3000=19033 - 6*3000=19033-18000=1033, not 33 mod 3000. $\ldots$	Qwen3-4B-TIES (30,160 tokens) Let’s test with exact fractions whether u 3 = u 1 . We have u 1 = 471/275 u 3 = (u 2 )/3 + 3/u 2 - 1 We computed u 2 = 57099 / 43175 Compute (u 2 )/3 = 19033 / 43175 3/u 2 = 3 $\times$ 43175 / 57099 = 129525 / 57099 = simplify: divide numerator and denominator by 3: 43175 / 19033 So u 3 = 19033 / 43175 + 43175 / 19033 - 1 Now compute this value and see if it equals u 1 = 471/275 Let’s compute the difference: Let A = 19033 / 43175 + 43175 / 19033 Then u 3 = A - 1 We need to check if A - 1 = 471/275 $\Rightarrow$ A = 471/275 + 1 = 746/275 Is 19033 / 43175 + 43175 / 19033 = 746/275? $\ldots$
Answer: $\boxed{57099/3=19033\Rightarrow m}$ $\times$	Answer: 748 $\times$
Qwen3-4B-MI-0.8 (27,562 tokens) Thus u 3 = - (19093/507) / (-131/13) = - (19093/507) * (-13/131) = (19093 * 13) / (507 * 131) Note that 507 = 31313, so 50713 = 313 2 13 = 313 3 ? Wait, 13507 = 13(500 + 7) = 6500 + 91 = 6591 But 19093 $\div$ 131: let’s check 131145 = 131(140 + 5) = 18340 + 655 = 18995; 19093 - 18995 = 98, not divisible. Wait, maybe I made a calculation error earlier. Wait, let’s step back. Maybe this substitution isn’t helping. Let’s go back to the a k and b k sequences with the recurrence: a k $+$ 1 = 3a k b k b k $+$ 1 = a k 2 - a k b k + b k 2 $\ldots$	Qwen3-4B-S 3 (18,362 tokens) So S k+1 = p k+1 + q k+1 = (p k 2 - p k q k + q k 2 )/3 + p k q k = (p k 2 - p k q k + q k 2 + 3 p k q k )/3 = (p k 2 + 2 p k q k + q k 2 )/3 = (p k + q k ) 2 / 3 Oh my goodness! That’s a beautiful simplification! S k+1 = (S k ) 2 / 3 Because p k 2 + 2p k q k + q k 2 = (p k + q k ) 2 , yes! So S k+1 = S k 2 / 3 That’s a huge breakthrough! Why didn’t I see this earlier?
Answer: 248 $\checkmark$	Answer: 248 $\checkmark$

Appendix

Table 10: AIME25, Example 1: A Reciprocal Recurrence (33.4% token reduction).

S^{3}

adds the two recurrences as soon as they are written in numerator/denominator form; Thinking derives the same pair and the trace ends inside the induction, and TIES substitutes to linearise the map instead.

Problem: Given that April $1^{\text{st }},2012$ fell on a Sunday, what is the next year in which April $1^{\text{st }}$ will fall on a Sunday?
Ground Truth: 2018
Qwen3-4B-Thinking (19,190 tokens) So why did I think maybe 2018 is too soon? Let’s see if we can find a reference in our mind. Wait, I recall that April 1, 2012 was a Sunday, and April 1, 2018 was a Sunday? Let’s think about historical events. Wait, 2012 Olympics were in London, April 1, 2012: I think that was a Sunday, yes (e.g., April 1, 2012 was a Sunday, as given). Then 2018: the Winter Olympics were in Pyeongchang, but April 1, 2018—maybe not helpful. $\ldots$	Qwen3-4B-TIES (17,533 tokens) So April 1, 2016 is a Sunday. But wait, the problem says April 1, 2012 is Sunday. Let’s compute: 2012 (leap) $\to$ shift 2 $\to$ 2013: Tuesday 2013 (not) $\to$ shift 1 $\to$ 2014: Wednesday 2014 (not) $\to$ shift 1 $\to$ 2015: Thursday 2015 (not) $\to$ shift 1 $\to$ 2016: Friday 2016 (leap) $\to$ shift 2 $\to$ 2017: Friday +2 = Sunday Wait, so 2017 is Sunday. But why does the shift depend on the current year? Yes: if the current year has a leap day, then the next year’s date is shifted by 2. So for April 1, Y to April 1, Y+1, the shift is 2 if Y is leap year, 1 otherwise. This is the correct rule. $\ldots$
Answer: 2018 $\checkmark$	Answer: 2017 $\times$
Qwen3-4B-MI-0.8 (26,918 tokens) Wait, let’s use an online calculator. I’ll pretend I can access one. According to timeanddate.com, April 1, 2012 was a Sunday. April 1, 2013: Monday April 1, 2014: Tuesday April 1, 2015: Wednesday April 1, 2016: Thursday April 1, 2017: Friday April 1, 2018: Saturday April 1, 2019: Sunday April 1, 2020: Monday April 1, 2021: Tuesday April 1, 2022: Wednesday April 1, 2023: Thursday April 1, 2024: Friday April 1, 2025: Saturday April 1, 2026: Sunday $\ldots$	Qwen3-4B-S 3 (5,804 tokens) Ah, right! Because from April 1st, 2015 to April 1st, 2016, the period covers February 29, 2016, which is a leap day, so it’s 366 days. So the key point is: when moving from April 1st of year Y to April 1st of year Y+1, the number of days is 366 if Y+1 is a leap year, otherwise 365. Wait, no. $\ldots$
Answer: 2029 $\times$	Answer: 2018 $\checkmark$

Appendix

Table 11: Olympiad-Bench, Example 2: When April 1st Is a Sunday Again (69.8% token reduction).

S^{3}

fixes which year owns the leap day before counting anything; TIES and MI re-derive that rule sentence by sentence.

Problem: Find the least positive integer $n$ for which there exists a set $\left\{s_{1},s_{2},\ldots,s_{n}\right\}$ consisting of $n$ distinct positive integers such that $\left(1-\frac{1}{s_{1}}\right)\left(1-\frac{1}{s_{2}}\right)\ldots\left(1-\frac{1}{s_{n}}\right)=\frac{51}{2010}$
Ground Truth: 39
Qwen3-30B-A3B-Thinking (29,074 tokens) However, exhaustive analysis shows that no combination of replacing elements in the set $\{1,2,...,39\}$ results in the exact target value due to constraints on integer solutions. - Thus, $n=39$ is not sufficient, and $n=40$ is the smallest possible	Qwen3-30B-A3B-TIES (21,702 tokens) We need this to be $\approx$ 1/1.0149 $\approx$ 0.9853. So [k(m+1)]/[m(k+1)] = 0.9853 $\Rightarrow$ k(m+1) = 0.9853m(k+1) $\Rightarrow$ km + k = 0.9853km + 0.9853m $\Rightarrow$ 0.0147km + k - 0.9853m = 0. For k=39: 0.014739m + 39 - 0.9853m = 0.5733m + 39 - 0.9853m = -0.412m + 39 = 0 $\Rightarrow$ m=39/0.412 $\approx$ 94.66. Try m=95: [3996]/[95*40]=3744/3800=0.98526 , which is very close to 0.9853. $\ldots$
Answer: 40 $\times$	Answer: Therefore, the minimal n is 40 $\times$
Qwen3-30B-A3B-MI-0.8 (29,974 tokens) So 445/429 > 335/323 > 364/351. Is there a k where (40/39)(k+1)/k = 335/323? Solve for k: (40/39)(k+1)/k = 335/323 $\Rightarrow$ (k+1)/k = (335/323)(39/40) = (33539)/(32340) = (567313)/(171985) = (67313)/(17198) = 2613/2584. So (k+1)/k = 2613/2584 $\Rightarrow$ k = 2584/(2613-2584) = 2584/29 $\approx$ 89.103, not integer. So closest integers are k=89 and k=90, but neither give the exact value.	Qwen3-30B-A3B-S 3 (8,513 tokens) Oh! That’s a key insight I missed earlier. So the product of (1 - 1/k) for k=2 to m is 1/m.
Answer: 39 $\checkmark$	Answer: 39 $\checkmark$

Appendix

Table 12: Olympiad-Bench, Example 3: A Product of Fractions Equal to 51/2010 (71.6% token reduction).

S^{3}

bounds the product by

1/(n+1)

and solves the resulting divisibility condition; Thinking and TIES search by approximation.

Problem: For a point $P,$ let $d_{1},$ $d_{2}$ and $d_{3}$ represents the distances from $P$ to the planes $x-z=0,$ $x-2y+z=0,$ and $x+y+z=0.$ Let $S$ be the set of points $P$ such that $d_{1}^{2}+d_{2}^{2}+d_{3}^{2}=36.$ Find the region of the volume enclosed by $S.$
Ground Truth: $288\pi$
Qwen3-4B-Thinking (16,803 tokens) Wait, wait! Wait a second—I just realized: in the problem statement, it says "the region of the volume enclosed by S". If S is the sphere (a surface), then the volume enclosed by S is the volume of the ball, but the problem says "the region", not "the volume". So they want the region (the set of points), which is the ball. In standard math notation, the set of points with x 2 + y 2 + z 2 $\leq$ 36 is the closed ball of radius 6 centered at the origin.	Qwen3-4B-TIES (8,188 tokens) Let me verify that again because that seems surprisingly clean. Let’s check each expansion again to make sure I didn’t make a mistake. First term: 3(x - z) 2 = 3x 2 - 6xz + 3z 2 Second term: (x - 2y + z) 2 = x 2 + (-2y) 2 + z 2 + 2(x(-2y) + xz + (-2y)z) = x 2 + 4y 2 + z 2 -4xy + 2xz -4yz $\ldots$
Answer: $\boxed{\text{solid sphere of radius }6}$ $\times$	Answer: $\boxed{288\pi}$ $\checkmark$
Qwen3-4B-MI-0.8 (9,979 tokens) Given that, I think the answer is the solid ball (closed ball) of radius 6 centered at the origin. To write it neatly in a box, maybe using standard notation. In some contexts, they might say "a sphere", but technically, the sphere is the boundary. Wait, but the problem says "the region of the volume enclosed by S", so S is the sphere (boundary), so enclosed volume is the ball.	Qwen3-4B-S 3 (4,772 tokens) Wow, all cross terms cancel out! That’s nice. So total = 6x 2 + 6y 2 + 6z 2 $\ldots$
Answer: $\boxed{x^{2}+y^{2}+z^{2}\leq 36}$ $\times$	Answer: $\boxed{288\pi}$ $\checkmark$

Appendix

Table 13: MATH-500, Example 4: Volume Enclosed by Three Distances (41.7% token reduction).

S^{3}

expands the sum once — the cross terms cancel and

x^{2}+y^{2}+z^{2}

falls out; Thinking and MI derive the same sphere and then answer with its name instead of its volume.

Problem: The binomial coefficient $\binom{n}{k}$ can be defined as the coefficient of $x^{k}$ in the expansion of $(1+x)^{n}$ . Similarly, define the trinomial coefficient $\binom{n}{k}_{3}$ as the coefficient of $x^{k}$ in the expansion of $(1+x+$ $\left.x^{2}\right)^{n}$ . Determine the number of integers $k$ with $0\leq k\leq 4048$ such that $\binom{2024}{k}_{3}\equiv 1\bmod 3$ .
Ground Truth: 648
Qwen3-4B-Thinking (28,852 tokens) Wait, is that right? Let’s list the multipliers again: i=0:1, i=1:3, i=2:3, i=3:3, i=4:1, i=5:1, i=6:3, i=7:1. Multiply step by step: 13=3; 33=9; 93=27; 271=27; 271=27; 273=81; 81*1=81. Yes, 81. But wait, let’s sanity check with smaller n to make sure our approach isn’t flawed. $\ldots$	Qwen3-4B-TIES (24,500 tokens) Number of valid choices for a m =2 is: - If there are s such digits, it’s 2 s-1 when s>0, and 1 when s=0. Wait for s=1: 2 0 =1 s=2:2 1 =2 s=0:1 Now let’s test: n=1: base3=1 $\to$ a 0 =1, s=0 $\to$ count=3 1 1=3 n=2: base3=2 $\to$ a 0 =2, s=1 $\to$ count=3 0 1=1 (actual is 2) $\ldots$
Answer: 648 $\checkmark$	Answer: 32 $\times$
Qwen3-4B-MI-0.8 (31,669 tokens) Reading remainders from bottom to top: $2024_{10}=2202222_{3}$ So the base-3 digits are: [2, 2, 0, 2, 2, 2, 2] Count: - Digits equal to 0: 1 - Digits equal to 1: 0 - Digits equal to 2: 6 $\ldots$	Qwen3-4B-S 3 (15,184 tokens) Let’s verify: $(x-1)^{2}=x^{2}-2x+1=x^{2}+x+1$ in $\mathbb{F}_{3}$ , since $-2\equiv 1\mod 3$ . Yes! Perfect. So in $\mathbb{F}_{3}[x]$ , $x^{2}+x+1=(x-1)^{2}$ . Therefore, $(1+x+x^{2})^{n}=(x-1)^{2n}$ in $\mathbb{F}_{3}[x]$ .
Answer: 64 $\times$	Answer: 648 $\checkmark$

Appendix

Table 14: CMIMC25, Example 5: Trinomial Coefficients Modulo 3 (47.4% token reduction).

S^{3}

substitutes

(1+x+x^{2})\equiv(x-1)^{2}

before applying Lucas; TIES and MI reduce the exponent by base-3 digits and then miscount the coefficients.

Problem: Call finite sets of integers $S$ and $T$ cousins if - $S$ and $T$ have the same number of elements, - $S$ and $T$ are disjoint, and - the elements of $S$ can be paired with the elements of $T$ so that the elements in each pair differ by exactly $1$ . For example, $\{1,2,5\}$ and $\{0,3,4\}$ are cousins. Suppose that the set $S$ has exactly $4040$ cousins. Find the least number of elements the set $S$ can have.
Ground Truth: 107
Qwen3-4B-Thinking (31,320 tokens) If we want 4040 cousins, which is 4040 = 2 3 * 5 * 101, but since the only possible cousin counts are powers of 2 (for good separation sets), the problem must have a typo, and the intended number is 4096 (2 12 ), so the minimal size is 12. But wait, no—another idea: in the problem, when S has a block of size 2, it "uses up" two elements to get 1 cousin, whereas a single element uses one element to get 2 cousins.	Qwen3-4B-TIES (24,345 tokens) Even if we try to reduce $C(S)$ by introducing conflicts (e.g., bringing two elements to distance 2), we reduce $C(S)$ by 1 per conflict. - To get $C(S)=4040$ from $2^{12}=4096$ , we’d need 56 such conflicts. - But each conflict requires two elements, so we’d need at least $2\times 56=112$ elements $\to$ $n\geq 112$ - This is much larger than 4039. Thus, no smaller $n$ can yield exactly 4040 cousins using any reasonable construction.
Answer: 12 $\times$	Answer: 4039 $\times$
Qwen3-4B-MI-0.8 (31,687 tokens) Wait, let’s use the k=2 example: Block 1: left=-1, right=1 Block 2: left=1, right=3 Configuration 1: (-1,1) – block1 left, block2 left Configuration 2: (-1,3) – block1 left, block2 right Configuration 3: (1,3) – block1 right, block2 right So: L(1)=1 (block1 left), R(1)=1 (block1 right) L(2)=1 (block2 left: only in config 1), R(2)=2 (block2 right: configs 2,3) Wait, no, L(2) is number of configs where block2 chooses left, which is 1; R(2)=2. $\ldots$	Qwen3-4B-S 3 (15,456 tokens) Yes! From the earlier examples: m=1: 2 = 1+1 m=2: 3 = 2+1 m=3: 4 = 3+1 And for a chain of length m, the valid assignments are sequences where once you choose right, all subsequent must choose right, so m+1 choices. Perfect! Therefore, if S is partitioned into k disjoint chains (each chain being a maximal set of elements spaced by 2), then the total number of cousins is the product of (length of chain + 1) over all chains.
Answer: 107 $\checkmark$	Answer: 107 $\checkmark$

Appendix

Table 15: AIME26, Example 6: Sets with Exactly 4040 Cousins (51.2% token reduction).

S^{3}

finds the multiplicative chain decomposition; Thinking and TIES reason from powers of two.

Problem: Let $a_{0},a_{1},a_{2},\ldots$ be a sequence of real numbers such that $a_{0}=0,a_{1}=1$ , and for every $n\geqslant 2$ there exists $1\leqslant k\leqslant n$ satisfying $a_{n}=\frac{a_{n-1}+\cdots+a_{n-k}}{k}$ Find the maximal possible value of $a_{2018}-a_{2017}$ .
Ground Truth: $\frac{2016}{2017^{2}}$
Qwen3-30B-A3B-Thinking (20,465 tokens) Similarly, in Path B, to get d4>0, we had a3 < a1 + a2? Wait, no, earlier for d4>0 with k=3 at n=4, we had a4=(a1+a2+a3)/3 >a3 $\Rightarrow$ a1+a2 >2a3. To maximize d4=(a4 -a3)= (a1+a2 -2a3)/3, need to minimize (a1+a2 -2a3), i.e., minimize a3 (since a1,a2 fixed once chosen). a1=1, a2 was minimized to 1/2 at n=2, then at n=3, to minimize a3, we take k=1 (a3=a2=1/2), so a3=1/2, hence d4=(1 + 1/2 - 2*(1/2))/3=(1 + 1/2 -1)/3=(1/2)/3=1/6, which is exactly what we got. And this is the maximum possible d4 because we minimized a3 (the variable term).	Qwen3-30B-A3B-TIES (31,161 tokens) For even $n$ , $a_{n}=2/3(1-1/2^{n})<2/3$ So $a_{2}=2/3(1-1/4)=1/2=0.5$ $a_{4}=2/3(1-1/16)=15/24=0.625$ $a_{6}=2/3(1-1/64)=63/96=0.65625$ So it’s increasing towards 2/3. So the even-indexed terms are increasing: $a_{2}=0.5$ , $a_{4}=0.625$ , $a_{6}=0.65625$ , …, approaching 2/3 $\approx$ 0.666…
Answer: $\boxed{\frac{1}{4034}}$ $\times$	Answer: 0 $\times$
Qwen3-30B-A3B-MI-0.8 (29,032 tokens) The best strategy to achieve a large difference at position $n$ is: - Keep the sequence constant at 1 from $a_{1}$ up to $a_{n-2}$ , - At step $n-1$ , compute $a_{n-1}=\frac{\text{sum of }a_{0}\text{ to }a_{n-2}}{n-1}$ , which gives $a_{n-1}=\frac{n-2}{n-1}$ , - Then at step $n$ , take $k=2$ : $a_{n}=\frac{a_{n-1}+a_{n-2}}{2}=\frac{(n-2)/(n-1)+1}{2}=\frac{2n-1}{2(n-1)}$ $\ldots$	Qwen3-30B-A3B-S 3 (13,098 tokens) Moreover, the only way to get a n =1 is if all the terms being averaged (the last k terms) are equal to 1, because if any term in the average is less than 1, the average would be less than 1.
Answer: $\boxed{\dfrac{1}{4034}}$ $\times$	Answer: $\boxed{Sothemaximalpossiblevalueis\ldots}$ $\checkmark$

Appendix

Table 16: Olympiad-Bench, Example 7: Maximising a Jump in a Moving Average (36.0% token reduction).

S^{3}

pins down when a term can equal 1, then takes the largest averaging window at the final step; the baselines take the smallest.

Problem: Let $f$ be the unique polynomial of degree at most $2026$ such that for all $n\in\{1,2,3,\ldots,2027\}$ , $f(n)=\begin{cases}1&\text{ if }n\text{ is a perfect square }\\ 0&\text{ otherwise }\end{cases}$ Suppose that $\frac{a}{b}$ is the coefficient of $x^{2025}$ in $f$ , where $a$ and $b$ are integers such that $\operatorname{gcd}(a,b)=1$ . Compute the unique integer $r$ between $0$ and $2026$ (inclusive) such that $a-rb$ is divisible by $2027$ . (Note that $2027$ is prime.)
Ground Truth: 1037
Qwen3-4B-Thinking (18,715 tokens) The leading coefficient $C$ of $f(x)$ satisfies: $C\equiv-m\mod p$ where $m$ is the number of perfect squares in $\{1,2,\ldots,p\}$ This result is verified through multiple test cases with small primes and confirmed via detailed analysis $\ldots$	Qwen3-4B-TIES (22,348 tokens) So $f(x)$ is degree at most 44? Wait, hold on! This is a critical point. The problem says "degree at most 2026", but if we have only 45 points where it’s 1 and others 0, isn’t the interpolating polynomial of degree at most 44? $\ldots$
Answer: 1982 $\times$	Answer: 0 $\times$
Qwen3-4B-MI-0.8 (13,991 tokens) Therefore: $c_{2025}\equiv-45\mod 2027.$ To get the representative in $[0,2026]$ , add 2027: $r=(-45)\mod 2027=2027-45=1982.$ — Final Answer $\ldots$	Qwen3-4B-S 3 (11,361 tokens) Differentiating both sides: $P^{\prime}(x)=2027x^{2026}-1\equiv-1\mod 2027$ So $P^{\prime}(n)=-1$ for all $n\in\mathbb{F}_{2027}$ .
Answer: 1982 $\times$	Answer: 1037 $\checkmark$

Appendix

Table 17: HMMT25, Example 8: A Polynomial Coefficient by Interpolation (18.8% token reduction).

S^{3}

exploits

P^{\prime}(x)\equiv-1

in the interpolation; the baselines replace the coefficient with a count of the non-zero values.

Problem: Find the smallest number $n$ such that there exist polynomials $f_{1},f_{2},\ldots,f_{n}$ with rational coefficients satisfying $x^{2}+7=f_{1}(x)^{2}+f_{2}(x)^{2}+\cdots+f_{n}(x)^{2}.$
Ground Truth: 5
Qwen3-30B-A3B-Thinking (31,456 tokens) I think I have to trust the theory here. The key points are: - n=1: impossible, as shown. - n=2: impossible, as shown (requires 7 to be sum of two rational squares, which it’s not). - n=3: impossible, as shown (7 cannot be sum of three rational squares). - n=4: possible, because 7 can be written as sum of four rational squares, and the orthogonality condition is solvable over Q (local conditions satisfied, so global solution exists by Hasse-Minkowski). Therefore, the minimal n is 4.	Qwen3-30B-A3B-TIES (15,776 tokens) Yes! Here’s a known example: $x^{2}+7=\left(\frac{x}{2}+\frac{3}{2}\right)^{2}+\left(\frac{x}{2}+\frac{3}{2}\right)^{2}+\left(\frac{x}{2}-\frac{3}{2}\right)^{2}+\left(\frac{x}{2}-\frac{1}{2}\right)^{2}.$ Let’s check: Each term: - $(\frac{x}{2}+\frac{3}{2})^{2}=\frac{x^{2}}{4}+\frac{3x}{2}+\frac{9}{4}$ , - Same for the second, - $(\frac{x}{2}-\frac{3}{2})^{2}=\frac{x^{2}}{4}-\frac{3x}{2}+\frac{9}{4}$ , - $(\frac{x}{2}-\frac{1}{2})^{2}=\frac{x^{2}}{4}-\frac{x}{2}+\frac{1}{4}$ . Sum: - $x^{2}$ : $4\times\frac{1}{4}=1$ , - $x$ : $\frac{3}{2}+\frac{3}{2}-\frac{3}{2}-\frac{1}{2}=(3-2)/2=1/2$ , - Constants: $2\times\frac{9}{4}+\frac{9}{4}+\frac{1}{4}=\frac{18+9+1}{4}=28/4=7$ . Still has $x$ term. But if we swap signs in the last two: Try: - $f_{1}=\frac{x}{2}+\frac{3}{2}$ , - $f_{2}=\frac{x}{2}+\frac{3}{2}$ , - $f_{3}=-\frac{x}{2}+\frac{3}{2}$ , - $f_{4}=-\frac{x}{2}+\frac{1}{2}$ . Then $x$ terms: $\frac{3}{2}+\frac{3}{2}+\frac{3}{2}+\frac{1}{2}=4$ . Worse. $\ldots$
Answer: 4 $\times$	Answer: 4 $\times$
Qwen3-30B-A3B-MI-0.8 (32,175 tokens) From the second equation: $ab=-cd\Rightarrow b=-\frac{c}{a}d$ (assuming $a\neq 0$ ). Substitute into the third equation: $b^{2}+d^{2}=\left(\frac{c^{2}}{a^{2}}+1\right)d^{2}=\frac{a^{2}+c^{2}}{a^{2}}d^{2}=\frac{1}{a^{2}}d^{2}.$ So: $\frac{1}{a^{2}}d^{2}+e^{2}+f^{2}=7.$ Let $u=\frac{d}{a}$ , then $u^{2}+e^{2}+f^{2}=7$ . This implies that 7 must be written as a sum of three rational squares, which is impossible (as shown earlier). Therefore, this configuration fails. Trying other configurations (e.g., three linear, one constant) similarly leads to contradictions or irrational coefficients $\ldots$	Qwen3-30B-A3B-S 3 (14,931 tokens) But Legendre’s three-square theorem says a natural number n is expressible as sum of three integer squares iff n is not of the form 4 a (8b + 7). 7 is exactly of the form 8b + 7 (b=0), so 7 cannot be written as sum of three integer squares. But can it be written as sum of four integer squares? Yes, by Lagrange’s four-square theorem, every natural number is sum of four integer squares. In fact, 7 = 2 2 + 1 2 + 1 2 + 1 2 = 4 + 1 + 1 + 1 = 7.
Answer: 5 $\checkmark$	Answer: 5 $\checkmark$

Appendix

Table 18: Olympiad-Bench, Example 9: Sum of Squares over the Rationals (53.6% token reduction).

S^{3}

reduces the question to how many rational squares 7 needs and answers with Legendre’s theorem; Thinking and TIES try to exhibit a construction instead.

Problem: Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end. Question: In the diagram of the food web shown, if the number of ferns decrease, the supply of salmon will most likely? Choices: (A) decrease (B) can’t tell (C) stay same (D) increase Ground Truth: decrease
Qwen3-VL-4B-Thinking (5,741 tokens) Wait, maybe the user made a mistake in the diagram description, but assuming the diagram is correct, we have to go with the connections given. Since the diagram shows Ferns connected to Rabbits, and Salmon connected to Insects: Ants (connected to Flowers), there’s no direct or indirect connection between Ferns and Salmon. Therefore, the supply of Salmon would stay the same. So the answer is C) stay same.	Qwen3-VL-4B-TIES (2,167 tokens) Therefore, we can’t tell. Wait, but let’s check again. The diagram has arrows from Ferns to Small Mammals: Rabbits. So rabbits eat ferns. Then, what do rabbits eat? If ferns are the main food, then fewer ferns $\to$ fewer rabbits. Now, do rabbits eat salmon? $\ldots$
Answer: C $\times$	Answer: B $\times$
Qwen3-VL-4B-MI-0.8 (2,705 tokens) Is there an arrow from Ferns to Salmon? Let me think. If the number of ferns decreases, what happens to salmon? If ferns are a food source for salmon, then less ferns would mean less food for salmon, so salmon population would decrease.	Qwen3-VL-4B-S 3 (280 tokens) Wait, in the diagram, there’s a line from ferns to salmon. So ferns are a food source for salmon. If ferns decrease, that means there’s less food available for salmon. So salmon’s supply would likely decrease because they have less food.
Answer: C $\times$	Answer: A $\checkmark$

Appendix

Table 19: MathVista-testmini, Example 10: Ferns and Salmon in a Food Web (87.1% token reduction).

S^{3}

follows the one edge the question turns on; the baselines reconstruct the whole food web from memory.

Problem: Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end. Question: What time does the clock show? Choices: (A) 9:30 (B) 1:30 (C) 4:30 (D) 5:30 (E) 11:30 Ground Truth: 4:30
Qwen3-VL-4B-Thinking (8,073 tokens) Wait, this is really hard without seeing the image. Wait, but the user provided a description: "the clock show" with numbers 1-12. The hour hand is pointing to 5. Or wait, no. $\ldots$	Qwen3-VL-4B-TIES (2,130 tokens) This is a bit tricky. Wait, perhaps the image shows the hour hand pointing to 5. But if the minute hand is at 6 (30 minutes), then the hour hand should be halfway between the current hour and the next. So if it’s 5:30, $\ldots$
Answer: D $\times$	Answer: D $\times$
Qwen3-VL-4B-MI-0.8 (4,041 tokens) Wait, the user provided a clock image. Let me try to imagine it. The clock has numbers 1 through 12. The minute hand is pointing at the 6 (which is 30 minutes), so the time is X:30.	Qwen3-VL-4B-S 3 (394 tokens) Looking at the image, the hour hand is pointing just past the 4, and the minute hand is pointing at the 6.
Answer: D $\times$	Answer: C $\checkmark$

Appendix

Table 20: MathVista-testmini, Example 11: Reading an Analogue Clock (81.5% token reduction).

S^{3}

reads both hands once; the baselines re-open the hour hand a dozen times.

Problem: Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end. Question: Quadrilateral $ABDC$ is a rectangle. If $m\angle 1=38$ , find $m\angle 2$ Choices: (A) 33 (B) 38 (C) 52 (D) 87 Ground Truth: 52
Qwen3-VL-2B-Thinking (16,634 tokens) The problem says quadrilateral ABDC is a rectangle, so ABDC is a rectangle, which means AB is parallel to CD, and AD is parallel to BC. Also, all angles in a rectangle are right angles, so angle at B is 90 degrees, right? Wait, the diagram shows a rectangle with points A, B, D, C, so the vertices are A, B, D, C, making ABDC a rectangle. So AB is one side, BD is a diagonal, and CD is the opposite side. Wait, no, in a rectangle, the order of the vertices matters. $\ldots$	Qwen3-VL-2B-TIES (7,072 tokens) First, let’s recall what we know about rectangles. A rectangle has all right angles, so angle at B is 90 degrees. Also, in a rectangle, opposite sides are equal, and diagonals are equal and bisect each other. Wait, but here we have quadrilateral ABDC as a rectangle. Wait, let’s check the diagram. First, let’s identify the rectangle. The diagram shows a rectangle with vertices A, B, D, C. Wait, the vertices are A, B, D, C. So ABDC is a rectangle. $\ldots$
Answer: B $\times$	Answer: C $\checkmark$
Qwen3-VL-2B-MI-0.8 (18,735 tokens) Wait, hold on, the vertices are A, B, D, C? Wait, the diagram shows a rectangle with points A, B, C, D. Let me check the labels. The rectangle is ABDC? Wait, no, the rectangle is probably ABCD, but the problem says ABDC. Wait, maybe it’s a typo, but let’s look at the diagram. The diagram has points A, B, C, D with a right angle at B, so AB is a side, BC is a side, CD is the other side, DA is the last side. Wait, the right angle is at B, so angle ABC is 90 degrees. That makes sense for a rectangle: all angles are 90 degrees, opposite sides equal. $\ldots$	Qwen3-VL-2B-S 3 (2,315 tokens) Now consider $\angle 2$ , which is at vertex $A$ , between side $AB$ and diagonal $AD$ . Since $AB$ and $AD$ are adjacent sides of the rectangle, $\angle BAD=90^{\circ}$ . We already have: $\angle BAC=38^{\circ}$ So: $\angle CAD=\angle BAD-\angle BAC=90^{\circ}-38^{\circ}=52^{\circ}$ Thus: $\angle 2=\angle CAD=52^{\circ}$ $\ldots$
Answer: B $\times$	Answer: C $\checkmark$

Appendix

Table 21: MathVista-testmini, Example 12: Which Angle Is Numbered Two (67.3% token reduction).

S^{3}

takes the two labels at

A

as the two parts of the right angle and subtracts once; Thinking and MI argue over the vertex order in “ABDC” and set angle 2 equal to angle 1.

Problem: Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end. Question: Are there fewer tandem bikes that are behind the brown metal bicycle than matte trucks on the left side of the green object? Choices: (A) Yes (B) No Ground Truth: No
Qwen3-VL-2B-Thinking (7,059 tokens) Let’s list all objects: 1. Small green bus (left side, gray and green) 2. Gray road bike (left of the truck) 3. Blue airplane (middle) 4. Small blue and green truck (green object? Wait, the green object is the truck with green color. Wait, the green truck is the one with the green color, so the truck is green. Then, the other truck is the gray pickup truck in the center. $\ldots$	Qwen3-VL-2B-TIES (1,478 tokens) Wait, "behind" in 3D space—relative to the brown bike. The brown bike is a road bike, and the tandem bike is a little to the right and maybe behind? Wait, no. Let’s visualize. $\ldots$
Answer: A $\times$	Answer: A $\times$
Qwen3-VL-2B-MI-0.8 (2,124 tokens) so behind the brown bicycle? Wait, no—position matters. The brown bicycle (road bike) is on the right side. The tandem bike is to the right of the brown bicycle, so maybe behind? Wait, no. Let’s look at the scene. $\ldots$	Qwen3-VL-2B-S 3 (551 tokens) Let’s see: the tandem bike is purple and located behind the brown bicycle. So that’s one tandem bike.
Answer: A $\times$	Answer: B $\checkmark$

Appendix

Table 22: MathVista-testmini, Example 13: Tandem Bikes and Matte Trucks (62.7% token reduction).

S^{3}

resolves “behind” once and compares; the baselines never settle the spatial relation.

Problem: Each of the following situations relates to a different company. [Image 1] For company D, find the missing amounts. A. $1,034,325B.$ 1,098,650 C. $1,081,584D.$ 1,210,732 Answer with the option letter only. Ground Truth: C
Qwen3-VL-4B-Thinking (10,382 tokens) Wait, maybe the question is asking for Revenues? Let’s see. Wait, let’s rearrange the formula for Revenues: Revenues = Net Income + Expenses - Gains + Losses Wait, let’s see. From Net Income = Revenues - Expenses + $\ldots$	Qwen3-VL-4B-TIES (6,003 tokens) Since all options satisfy the equation, but the problem must have one correct answer, maybe I made a mistake in the problem $\ldots$
Answer: D $\times$	Answer: A $\times$
Qwen3-VL-4B-MI-0.8 (8,611 tokens) So all options work. But that can’t be. There must be a mistake in the problem or my reasoning. Wait, maybe the question is to find the missing amount for Company D, which is Expenses, and the $\ldots$	Qwen3-VL-4B-S 3 (1,463 tokens) Wait, so the only missing amount for Company D is Expenses. Let’s verify. Net Income = Revenues - Expenses + Gains - Losses We can rearrange to solve for Expenses: Expenses = Revenues + Gains - Losses
Answer: A $\times$	Answer: C $\checkmark$

Appendix

Table 23: MMMU, Example 14: A Missing Figure in a Trial Balance (75.6% token reduction).

S^{3}

identifies which cell is missing; the baselines solve for a cell that was already given.

Problem: Use the image to analyze the effect of the following on the level of real interest rates:[Image 1] a. Businesses become more pessimistic about future demand for their products and decide to reduce their capital spending. b. Households are induced to save more because of increased uncertainty about their future Social Security benefits. c. The Federal Reserve Board undertakes open-market purchases of U.S. Treasury securities in order to increase the supply of money. A. a.increase ;b.decrease ;c.increase B. a.increase ;b.decrease ;c.decrease C. a.decrease ;b.decrease ;c.decrease D. a.decrease ;b.increase ;c.decrease Answer with the option letter only. Ground Truth: C
Qwen3-VL-2B-Thinking (7,620 tokens) So for option a: demand for funds (for investment) decreases $\to$ demand curve shifts left $\to$ intersection is to the left of E $\to$ interest rate is lower (decrease). Option b: households save more $\to$ demand for funds (for investment) decreases $\to$ demand curve shifts left $\to$ interest rate is lower (decrease). Option c: Federal Reserve buys Treasury securities $\to$ supply of money increases $\to$ supply curve shifts right $\to$ intersection is to the right of E $\to$ interest rate is lower (decrease). So all three options decrease the interest rate. $\ldots$	Qwen3-VL-2B-TIES (919 tokens) D. a.decrease ;b.increase ;c.decrease $\to$ yes, that’s D. Wait, let me confirm again. For a: businesses reduce capital spending $\to$ demand for funds decreases $\to$ demand curve shifts left $\to$ equilibrium real rate decreases $\to$ correct. For b: households save more $\to$ demand for funds increases $\to$ demand curve shifts right $\to$ equilibrium real rate increases $\to$ correct. For c: Fed buys Treasury securities $\to$ increases money supply $\to$ supply of funds shifts right $\to$ equilibrium real rate decreases $\to$ correct. So the answer is D. $\ldots$
Answer: C $\checkmark$	Answer: D $\times$
Qwen3-VL-2B-MI-0.8 (1,023 tokens) In the supply and demand model for funds, the demand curve is the demand for funds by businesses, households, etc. If businesses reduce their capital spending, that’s a decrease in the demand for funds (since they’re not borrowing as much to spend on capital). So the demand curve shifts left. The intersection of supply and demand shifts left, so the equilibrium real rate of interest increases. Wait, but in the graph, the supply curve is upward sloping, and demand is downward sloping. $\ldots$	Qwen3-VL-2B-S 3 (341 tokens) So a is decrease. b. Households save more due to uncertainty about Social Security benefits. Saving more means more funds in the banking system, so the supply of funds increases. Supply of funds shifts right. That would lead to a lower real interest rate. Because with more supply, the equilibrium interest rate falls. So b is decrease. c. Federal Reserve buys Treasury securities, increasing money supply. That shifts the supply curve for funds to the right (since more money is in the economy). So supply increases, equilibrium interest rate decreases. So c is decrease. $\ldots$
Answer: B $\times$	Answer: C $\checkmark$

Appendix

Table 24: MMMU, Example 15: Shifts in a Loanable-Funds Diagram (95.5% token reduction).

S^{3}

moves one curve per shock and reads the new crossing off the diagram; MI sends the fall in capital spending the wrong way and TIES moves the rise in saving onto the demand curve.

Problem: What is the most likely profession of the first speaker?
Audio transcript (reader aid; the model received only the audio) : “all right bro looking good we’re done yeah we’re all done man what’s up there you go this is for you you want me to eat this yeah man candy yeah sure all right that was a test dude which you failed”
Ground Truth: Dentist
Qwen3-Omni-30B-A3B-Thinking (1,978 tokens) The key part is that they’re talking about candy, and then one says it was a test that the other failed. But what does this have to do with professions? Now, the question is: What is the most likely profession of the first speaker? Options are Dentist, Teacher, Fitness Coach, Nurse. I need to think about contexts where someone might give candy as a test or something related to health. Let me consider each option: - Dentist: Dentists often talk about oral health, and candy might be related to teeth. But dentists usually advise against eating candy because it’s bad for teeth. However, in this case, the first speaker gives candy and says it was a test. Maybe it’s a test for something like how people react to candy, but that seems odd.	Qwen3-Omni-30B-A3B-TIES (549 tokens) The first speaker says, ’All right, bro. Looking good. We’re done?’ The second speaker asks, ’We’re done?’ and the first speaker confirms, ’Yeah, we’re all done, man.’ The second speaker then says, ’What’s up?’ and the first speaker responds, ’There you go. This is for you.’ The second speaker asks, ’You want me to eat this?’ and the first speaker says, ’Yeah, man. Why not? The candy. $\ldots$
Answer: C $\times$	Answer: C $\times$
Qwen3-Omni-30B-A3B-MI-0.8 (7,446 tokens) Perhaps for a dentist, after a check-up, they might say "Looking good," and then give a small treat, but candy is bad for teeth, so it doesn’t make sense for a test. Unless it’s to see if the patient resists, but the patient is given candy to eat, so if they eat it, they fail, but dentists don’t usually do that. For a teacher, it’s unlikely. For a nurse, similar. Let’s think about the word "test." In medical contexts, there are glucose tests, but candy isn’t used for that.	Qwen3-Omni-30B-A3B-S 3 (316 tokens) Crucially, the word ’test’ is used in what appears to be a professional context, suggesting a formal evaluation.
Answer: C $\times$	Answer: A $\checkmark$

Appendix

Table 25: MMAR, Example 16: A Profession Inferred from a Conversation (42.4% token reduction).

S^{3}

takes the word “test” as the domain cue; the baselines supply a fitness practice that does not exist.

Problem: Which speaker ate the powdered donut
Audio transcript (reader aid; the model received only the audio) : “Hey, wait my last powder donut it wasn’t me wasn’t me Guys that was his last powder donut. Don’t be so disrespectful. Okay?”
Ground Truth: Fourth person
Qwen3-Omni-30B-A3B-Thinking (3,929 tokens) Here, "his" refers to Speaker 1 (the owner), confirming the donut was eaten, and the phrase "Don’t be so disrespectful" implies Speaker 4 is addressing the eater. The coughing directly after Speaker 3’s denial suggests they are the one who ate the donut, as the physical reaction gives away their guilt. Speaker 1 is the owner and not the eater, Speaker 2 denied responsibility without coughing, and Speaker 4 is accusing others, so they are not the eater. Therefore, the third speaker (Speaker 3) ate the powdered donut.	Qwen3-Omni-30B-A3B-TIES (484 tokens) Speaker 2 (female) says, ’It wasn’t me,’ which is a direct denial of eating it. Speaker 1 then says, ’Wasn’t me,’ which is another denial. After these denials, Speaker 1 starts coughing, and then says, ’Guys, that was his last powdered donut. Don’t be so disrespectful. Okay, fresh up right now.’ This is the key part. $\ldots$
Answer: B $\times$	Answer: A $\times$
Qwen3-Omni-30B-A3B-MI-0.8 (3,006 tokens) The audio: "Wasn’t me." then "cough cough cough Oh!" then "Guys, that was his last powdered donut…" So coughing is separate from Speaker 4’s speech. Therefore, the coughing is from one of the previous speakers. Since Speaker 3 just said "Wasn’t me," and then coughs, it’s likely Speaker 3. Speaker 2 spoke earlier, so less likely. Therefore, Speaker 3 ate it.	Qwen3-Omni-30B-A3B-S 3 (218 tokens) Then, analyzing the audio reveals four speakers: Speaker 1 (first person) asks, "Hey, when the last powdered donut?", Speaker 2 (second person) denies, "It wasn’t me.", Speaker 3 (third person) denies, "Wasn’t me.", and Speaker 4 (fourth person) admits by saying, "Yeah, that was me.", followed by coughing and gagging, then clarifying, "That was his last powdered donut. Don’t be so disrespectful." This admission occurs when the speaker identifies themselves as the one who ate it.
Answer: B $\times$	Answer: D $\checkmark$

Appendix

Table 26: MMAR, Example 17: Who Ate the Powdered Donut (55.0% token reduction).

S^{3}

reads the fourth speaker’s admission; the baselines infer the culprit from a cough.

Problem: What ingredients did Speaker 1 say she used in the trifle?
Audio transcript (reader aid; the model received only the audio) : “it’s a trifle it’s got all of these layers it’s got all of these layers first there’s a layer of ladyfingers then a layer of jam then custard which i made from scratch then raspberries more ladyfingers then beef sauteed with peas and onions and then bananas and then i just put some whipped cream on top it tastes like feet i like it are you kidding”
Ground Truth: (D) Ladyfingers, jam, custard, raspberries, beef, peas, onions, bananas, whipped cream
Qwen3-Omni-30B-A3B-Thinking (1,373 tokens) A. Bread, jelly, pudding, strawberries, chicken, carrots, garlic, apples, cream - Bread? Not mentioned; she said lady fingers, which are like cookies or biscuits, but not bread. - Jelly? She said jam, which is similar, but option says jelly. - Pudding? She said custard, which might be confused, but custard is different. - Strawberries? She said raspberries. - Chicken? She said beef. - Carrots, garlic, apples? Not mentioned. - Cream? She said whipped cream, but option says cream, which might be close. $\ldots$	Qwen3-Omni-30B-A3B-TIES (589 tokens) Option A includes bread, jelly, pudding, strawberries, chicken, carrots, garlic, apples, cream. ’Bread’ is not correct as ’ladyfingers’ were mentioned; ’jelly’ is not correct as ’jam’ was used; ’strawberries’ is incorrect as ’raspberries’ were stated; ’chicken’ is wrong because ’beef’ was used; and other items like ’carrots’ and ’garlic’ were not mentioned, so A is incorrect. $\ldots$
Answer: D $\checkmark$	Answer: D $\checkmark$
Qwen3-Omni-30B-A3B-MI-0.8 (1,728 tokens) Option A: Bread, jelly, pudding, strawberries, chicken, carrots, garlic, apples, cream - Bread: she said lady fingers, which are a type of cookie or biscuit, not bread. But lady fingers are often used in trifles, so maybe similar, but not exactly bread. - Jelly: she said jam, which is similar, but jelly is usually fruit spread, jam is thicker. But she said jam. - Pudding: she said custard, which is different. Pudding is usually set with gelatin or cornstarch, custard is egg-based. Not the same. - Strawberries: she said raspberries, not strawberries. $\ldots$	Qwen3-Omni-30B-A3B-S 3 (238 tokens) Next, I compared these ingredients with each option: Option A includes bread, strawberries, chicken, carrots, and garlic which were not mentioned; Option B includes cake, marmalade, cherries, turkey, corn, peppers, and melons which were not mentioned; Option C includes cookies, blueberries, pork, beans, tomatoes, peaches, and yogurt which were not mentioned; Option D lists ladyfingers, jam, custard, raspberries, beef, peas, onions, bananas, and whipped cream which exactly matches Speaker 1’s description. Therefore, the correct answer is D. $\ldots$
Answer: D $\checkmark$	Answer: D $\checkmark$

Appendix

Table 27: MMAU_testmini, Example 18: The Ingredients of a Trifle (59.6% token reduction).

S^{3}

recalls the layers once and dismisses each wrong list in a clause; the baselines re-check every distractor ingredient by ingredient.

Problem: How many different Chinese tones are involved in the six Chinese pronunciation words demonstrated by the speaker?
Audio transcript (reader aid; the model received only the audio) : “Chinese is easy when you learn four tones in this way. Move your head with the tone shape. Try this.”
Ground Truth: 4
Qwen3-Omni-30B-A3B-Thinking (4,147 tokens) Each syllable’s tone is as follows: - First "m $\bar{a}$ " in "mama": first tone ( $\bar{a}$ ) - Second "m $\bar{a}$ " in "mama": first tone ( $\bar{a}$ ) - "q $\acute{\imath}$ ": second tone ( $\acute{a}$ ) - "m $\check{a}$ " in "q $\acute{\imath}$ m $\check{a}$ ": third tone ( $\check{a}$ ) - "m $\check{a}$ " in "m $\check{a}$ m $\bar{a}$ ": third tone ( $\check{a}$ ) - "m $\bar{a}$ " in "m $\check{a}$ m $\bar{a}$ ": first tone ( $\bar{a}$ ) The tones used are first, second, and third. The fourth tone ( $\grave{a}$ ) is not demonstrated in any of the six syllables. Therefore, there are t $\ldots$	Qwen3-Omni-30B-A3B-TIES (1,841 tokens) 3. m $\check{a}$ m $\bar{a}$ - "m $\check{a}$ " is third tone. - "m $\bar{a}$ " is first tone. Summary of Tones Used: - First tone: appears in "m $\bar{a}$ " (repeatedly). - Second tone: appears in "q $\acute{\imath}$ ". - Third tone: appears in "m $\check{a}$ " (repeatedly). The fourth tone is not present in any of the six syllables.
Answer: A $\times$	Answer: A $\times$
Qwen3-Omni-30B-A3B-MI-0.8 (5,912 tokens) She says: " $\bar{a}$ , $\acute{a}$ , $\check{a}$ , $\grave{a}$ " – that’s the four tones. Then she says "Try this: m $\bar{a}$ , m $\bar{a}$ , q $\acute{\imath}$ , m $\check{a}$ , m $\check{a}$ , m $\check{a}$ " – let’s parse that. She says: "m $\bar{a}$ , m $\bar{a}$ , q $\acute{\imath}$ , m $\check{a}$ , m $\check{a}$ , m $\check{a}$ " – but that might be a typo or mispronunciation. Looking back: "m $\bar{a}$ , m $\bar{a}$ , q $\acute{\imath}$ , m $\check{a}$ , m $\check{a}$ , m $\check{a}$ " – but "m $\bar{a}$ " is repeated, and "q $\acute{\imath}$ " is different. Let me read the audio transcript: "Try this: m $\bar{a}$ , m $\bar{a}$ , q $\acute{\imath}$ , m $\check{a}$ , m $\check{a}$ , m $\check{a}$ " – but it says "m $\bar{a}$ m $\bar{a}$ q $\acute{\imath}$ m $\check{a}$ m $\check{a}$ m $\check{a}$ " – that seems odd. Perhaps it’s "m $\bar{a}$ , m $\bar{a}$ , q $\acute{\imath}$ , m $\check{a}$ , m $\check{a}$ , m $\check{a}$ " meaning six words. But "m $\bar{a}$ " is first tone, "q $\acute{\imath}$ " is second tone, "m $\check{a}$ " is third tone, and "m $\grave{a}$ " might be fourth tone, but she says "m $\check{a}$ " multiple times. Let’s look at the audio: she says "m $\bar{a}$ , m $\bar{a}$ , q $\acute{\imath}$ , m $\check{a}$ , m $\check{a}$ , m $\check{a}$ " – but "m $\check{a}$ " is third tone, and she says it three times? That doesn’t make sense for different tones.	Qwen3-Omni-30B-A3B-S 3 (535 tokens) Then in the examples: - m $\bar{a}$ m $\bar{a}$ : both tone 1 - q $\acute{\imath}$ m $\check{a}$ : q $\acute{\imath}$ is tone 2, m $\check{a}$ is tone 3 - m $\check{a}$ m $\acute{a}$ : m $\check{a}$ is tone 3, m $\acute{a}$ is tone 4 So the tones used are tone 1, tone 2, tone 3, tone 4. All four tones are demonstrated.
Answer: A $\times$	Answer: D $\checkmark$

Appendix

Table 28: MMAR, Example 19: Counting the Chinese Tones Demonstrated (70.9% token reduction).

S^{3}

transcribes the last syllable as “má”; the baselines hear “mă” and lose the fourth tone.

Problem: Which types of disfluencies are present in the audio? - Filled pauses: e.g., uh, um - Discourse markers: e.g., well, you know - Restarts: interrupted or repeated sentence starts - Explicit editing terms: e.g., I mean - None: if the speech is fluent.
Audio transcript (reader aid; the model received only the audio) : “Well, my sister is the seamstress of the two of us, but she doesn’t China paint, so we both have our expertise side.”
Ground Truth: discourse markers, restarts
Qwen3-Omni-30B-A3B-Thinking (466 tokens) Then, the audio starts with a short, sharp laugh: "Heh." This is a common filled pause, often used to indicate a brief hesitation or as a conversational filler. Next, the speaker says: "Well my sister is the seamstress of the two of us…" The word "Well" is a classic discourse marker used to introduce a statement.	Qwen3-Omni-30B-A3B-TIES (1,459 tokens) First, listen to the speech: "Huh, well my sister is the sinstrous of the two of us. And but she doesn’t chain up paint, so we both have our expertise site." Now, break down each type: 1. Filled pauses: The speaker says "Huh" at the beginning. "Huh" is a filled pause, similar to "um" or "uh", used to fill a moment of hesitation.
Answer: D $\times$	Answer: D $\times$
Qwen3-Omni-30B-A3B-MI-0.8 (1,944 tokens) Then, analyzing the audio content: the speaker says, ’Huh. Well, my sister is the seamstress of the two of us, and but she doesn’t China paint, so we both have our expertise side.’ Specifically, at the very beginning, there’s a ’Huh’ which is a filled pause (a short, non-lexical sound expressing hesitation or thought), and around 00:03, the speaker says ’and but’ which is a classic example of a restart $\ldots$	Qwen3-Omni-30B-A3B-S 3 (271 tokens) First, listen to the beginning: "Ha, well my sister is the simstrous of the two of us…" - The initial "Ha" is a chuckle, which is a filler but not a standard filled pause like "uh" or "um". - The word "well" is a discourse marker, used to introduce the sentence. - The phrase "simstrous" is a self-correction for "simstress". This is a restart/repetition of a word.
Answer: D $\times$	Answer: A $\checkmark$

Appendix

Table 29: MMSU, Example 20: Types of Disfluency in Speech (41.8% token reduction).

S^{3}

hears the opening sound as a chuckle; the baselines count it as a filled pause.

Explore similar work

Sep 28, 2026cs.AI

Efficient Reasoning via Constrained Optimization in Latent Space

Large Reasoning Models (LRMs) have shown remarkable reasoning capabilities, yet they still suffer from overthinking, generating redundant reasoning steps which incur substantial token consumption. Existing methods, such as suppressing reflective keywords or forcing shorter reasoning lengths, attempt to mitigate this issue but inevitably truncate necessary steps and induce underthinking, thereby compromising performance. To address this dilemma, we investigate the latent representations and observe that efficient reasoning steps naturally cluster into a concentrated region in latent space, while those deviating from this region tend to produce verbose sequences. To leverage this, we keep reasoning focused within this region via a quadratic program which projects deviating hidden states back into the region. Then we propose a novel training-free framework to achieve efficient reasoning that reduces token generation costs without sacrificing performance. Extensive experiments conducted on four models ranging from 1.5B to 14B, and across six benchmarks in math reasoning, coding, and scientific QA, validate the effectiveness of our method, up to a 12.1% improvement in accuracy while reducing generated tokens by 11.8% to 52.8%. Codes are available at \href{https://github.com/hzn18/Opt4Reasoning}{https://github.com/hzn18/Opt4Reasoning}.

Zhinan Hou, XingChen Li, Keyou You

Tsinghua University

Large Reasoning Models Reasoning Traces Latent Space Constrained Optimization

Ask PDF

May 31, 2026cs.LG

ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks

Large language models often improve on difficult tasks by spending inference-time compute on a reasoning trace before producing the final answer. That extra computation can be useful, but it also raises latency, token cost, and deployment complexity. We introduce \textbf{ThinkSwitch}, a low-compute procedure for co-training paired instruct and thinking checkpoints. Starting from compatible Qwen3-4B instruct and thinking models, each iteration asks the thinking checkpoint to generate answers, removes the reasoning trace, distills the answer-only pairs into the instruct checkpoint with QLoRA, and reconstructs a thinking checkpoint with spherical weight interpolation. The only human-supplied inputs are task prompts; the labels are generated by the model itself. On a 30-question AIME 2026 evaluation, ThinkSwitch improves the instruct checkpoint from 10/30 to 20/30 and the thinking checkpoint from 14/30 to 22/30. On a 30-question PubMedQA subset, it improves the instruct checkpoint from 13/30 to 18/30 and the thinking checkpoint from 18/30 to 25/30. The complete experiment uses 15 training prompts per domain and costs $2.86 on a single cloud RTX 3070. The results are small-scale, but they indicate that targeted distillation loops can move part of the benefit of explicit reasoning into weights while preserving a separate thinking mode.

Dhruv Saini, Rohan Pandey

Bellevue High School · DigitalOcean

Reasoning Skills Dataset Distillation Inference-Time Intermediate Checkpoints Interpolation Smart Homes

Ask PDF

Jun 2, 2026cs.CL

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

Extended chain-of-thought (CoT) traces improve LLM reasoning but incur substantial computational and memory costs. While existing CoT compression methods mitigate this by condensing thought steps into compact representations via memory tokens and retaining only these representations at inference time, the loss of fine-grained information makes subsequent steps more error-prone. To alleviate this, we propose \textbf{HybridThinker}, where in addition to preserved these representations, thought steps are also temporarily retained to provide fine-grained details. However, we observe that naively keeping thought steps accessible to subsequent steps \emph{during training} lets the model bypass memory tokens by retrieving information directly from these steps, leaving the model's ability to compress and retrieve information through memory tokens insufficiently trained. We therefore introduce a hybrid training scheme, in which only some thought steps are directly accessible through attention to subsequent steps, while the other thought steps are masked, forcing the model to use memory tokens for compression and retrieval. Across 4 reasoning benchmarks, HybridThinker matches the uncompressed baseline, advancing the state of the art in CoT compression by 5.8 points on average accuracy with similar inference time. Ablation studies confirm that both temporary thought-step retention and the hybrid training scheme contribute to these gains.

Xin Liu, Runsong Zhao, Xinyu Liu +8

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China · China Unicom Cloud-Link, China · NiuTrans Research, Shenyang, China

Chain-of-Thought Reasoning Latent Thoughts Peak Memory

Ask PDF