Shorter Is Not Cheaper: A Controlled Comparison of Declarative Animation Specifications and Free-Form Code for LLM-Generated Explanations

Jiantang Huang

cs.CLOct 23, 2026

Shorter Is Not Cheaper: A Controlled Comparison of Declarative Animation Specifications and Free-Form Code for LLM-Generated Explanations

Authors: Jiantang Huang

Abstract

Chat assistants increasingly answer conceptual questions with interactive, animated explanations, generated either as free-form HTML/JavaScript or as declarative specifications that a fixed runtime plays. The declarative route is widely expected to be cheaper, faster and more reliable, because its outputs are shorter and constrained. We test this expectation in controlled comparisons whose hypotheses and analysis plans were frozen before the runs. We designed ExplainSpec, a time-indexed declarative animation format, and an observation contract that makes every generated explanation programmatically inspectable; validated, pacing-agnostic checkers score 20 explanation requests from five domains. Three Claude models (360 jobs) and, in a follow-up, the open-weight Qwen3-14B with thinking enabled and disabled (240 jobs) answered every request in both formats. For every model the specifications were 25-48% shorter in answer tokens (output excluding reasoning), yet with reasoning enabled total output fell significantly only for the largest Claude model and no model generated significantly faster. The reasons differed. The Claude models reasoned far more before writing specifications (exploratory; for the two larger models, from 15-20% to 85-88% of generations). Qwen3-14B reasoned about as much in both formats (a pre-registered test of this shift was not supported), and its shorter answers saved little against a reasoning trace more than twice as long. Nor were specifications more reliable: free-form pages were valid after one repair round in 95-100% of jobs for every model, whereas Qwen3-14B produced valid specifications in only 33% of jobs and passed all semantic tests in 17% (HTML/JavaScript: 62%). For reasoning models, a shorter format alone therefore does not make generation cheaper, faster or more reliable, and a format learned in context can cost reliability.

Figures & tables

Domain	Task	Parameters	Required state variables	Settings
Physics	Projectile motion	v0, theta	t, x, y	3
Physics	Simple pendulum	L, theta0	t, angle	3
Physics	Damped spring-mass oscillator	k, c	t, x	3
Physics	RC circuit charging	R, C	t, v_c, i	3
Algorithms	Binary search	target	lo, hi, mid, found	3
Algorithms	Quicksort partition (Lomuto)	variant	arr, j, pivot	3

Table 1: The 20 tasks. Each task is tested at its default and 1–2 further parameter settings (“Settings”).

Model	Format	1st	repair	1st	repair	total	answer	reasoning	jobs	(s)	1st	repair
		Valid (%)		All tests pass (%)		Output tokens (median)			Reasoning	Latency	Cost ($, 60 jobs)
Opus	HTML/JS	100	100	100	100	2,806	2,795	0	9	21.1	4.43	4.43
	Spec	88	100	83	95	2,472	2,043	402	51	20.7	3.89	4.63
Sonnet	HTML/JS	100	100	95	95	2,595	2,558	0	12	13.6	2.13	2.13
	Spec	95	100	85	90	2,886	1,822	933	53	18.4	2.16	2.32
Haiku	HTML/JS	100	100	93	93	5,538	2,926	2,454	56	21.6	0.20	0.20

Table 2: Outcomes by model and format (60 jobs each). Token and latency columns are medians over first attempts (latency is generation time); answer tokens exclude reasoning tokens, which the interface reports as “thinking tokens”; reasoning jobs: first attempts with any reasoning (of 60). Cost is the total for the 60 jobs, for first attempts only and including repair rounds.

Measure	Effect	Opus	Sonnet	Haiku
H1: total output tokens	median ratio Spec/HTML	0.81 [0.76, 0.93]	0.99 [0.89, 1.16]	1.13 [0.96, 1.70]
	Holm-adjusted $p$	0.040	1.000	1.000
H1: generation latency	median ratio Spec/HTML	0.90 [0.80, 1.01]	1.23 [0.96, 1.37]	1.25 [1.04, 1.87]
	Holm-adjusted $p$	0.368	1.000	1.000
H2: valid (with repair)	mean difference (pp)	+0 [+0, +0]	+0 [+0, +0]	$-$ 3 [ $-$ 8, +0]
	Holm-adjusted $p$	–	–	1.000

Table 3: Pre-registered tests on 20 paired tasks. Brackets: 95% bootstrap CIs. H1 and the secondary measure use first attempts, H2 and H3 the results after repair. Exact Wilcoxon signed-rank tests, one-sided in the hypothesised direction for H1, H2 and the secondary measure and two-sided for H3, with Holm adjustment across the three models; “–”: all differences are zero, so no test is possible; pp: percentage points.

Figure 2: Median per-task ratio Spec / HTML/JS with 95% bootstrap CIs (log scale; dashed line: no difference). Specifications are consistently shorter in answer tokens; total output, which includes reasoning, falls only for Opus, and generation latency is not significantly reduced for any model.

Figure 3: Validity and semantic pass rates by model and format, on the first attempt and after the single repair round (frozen protocol).

Category	HTML/JS	Spec
Wrong content (wrong algorithm step, early stop, wrong update)	1 (1)	4 (4)
Still invalid after the repair round	0 (0)	2 (2)
Extra leading state with index 0 (brief: index starts at 1)	0 (0)	8 (2)
Binary search: mid = null between probes (brief: only before the first)	3 (0)	0 (0)
Go-Back-N: sent grows within a round (ambiguity in the brief)	3 (0)	2 (2)
Harness asymmetry (runtime error swallowed, no repair; section 5.4 )	0 (0)	3 (0)

Table 4: Classification of the 26 jobs that do not pass all semantic tests after repair (frozen protocol). Each job is counted in the first matching category. In brackets: jobs that still fail after the correction and with relaxed checkers ( section 5.4 ), that is, jobs with errors beyond the strict readings and the harness asymmetry.

Analysis	Opus	Sonnet	Haiku
Primary (frozen protocol)	100 vs 95	95 vs 90	93 vs 83
Runtime errors treated as exceptions (corrected)	100 vs 97	95 vs 93	93 vs 83
Corrected, with relaxed checkers (exploratory)	100 vs 98	98 vs 97	100 vs 88

Table 5: Jobs passing all semantic tests after repair (%, HTML/JS vs Spec) under the frozen protocol, after treating runtime errors like exceptions (corrected), and additionally with relaxed checkers (exploratory).

Error raised by the runtime or validator	Artifacts
Object indexed with a dynamic key / member access on a non-object	8
Array function applied to a non-array (e.g. a null initial state)	4
= prefix inside a plain expression field ( dur , derived )	4
Malformed expression (unbalanced brackets, unquoted text)	3
Non-terminating while loop	2
Derived variable used inside a step	2

Table 6: Validator and runtime errors in Spec artifacts (all attempts, first error per artifact).

Mode	Format	1st	repair	1st	repair	total	answer	reasoning	(s)
		Valid (%)		All tests pass (%)		Output tokens (median)			Latency
Thinking	HTML/JS	85	95	60	62	4,655	1,559	3,370	103.7
	Spec	13	33	8	17	4,513	906	3,686	95.0
No thinking	HTML/JS	83	97	35	40	1,618	1,618	0	29.2
	Spec	10	32	0	3	982	982	0	18.1

Table 7: Qwen3-14B follow-up: outcomes by mode and format (60 jobs each), scored with the corrected harness. Token and latency columns are medians over first attempts; reasoning tokens are counted from the streamed thinking; latency is measured on one local GPU.

Test	Direction	Effect [95% CI]	$p$	Holm $p$
Q1 (primary): reasoning tokens	Spec $>$ HTML/JS	1.07 [0.95, 1.33]	0.071	0.286
Q2a: answer tokens	Spec $<$ HTML/JS	0.52 [0.49, 0.61]	$<$ 0.001	$<$ 0.001
Q2b (H1): total output tokens	Spec $<$ HTML/JS	0.87 [0.76, 1.04]	0.184	0.552
Q2c (H1): generation latency	Spec $<$ HTML/JS	0.89 [0.77, 1.11]	0.324	0.648
Q3 (H2): valid (with repair), pp	Spec $>$ HTML/JS	$-$ 62 [ $-$ 78, $-$ 43]	1.000	1.000
Q4 (H3): all tests pass (with repair), pp	two-sided	$-$ 45 [ $-$ 63, $-$ 27]	$<$ 0.001	0.004

Table 8: Qwen3-14B follow-up: pre-registered tests with thinking enabled (one Holm family of six tests) and the exploratory ablation. Effects are median per-task ratios Spec/HTML/JS for tokens and latency and mean differences in percentage points (pp) for rates, with 95% bootstrap CIs; exact Wilcoxon signed-rank tests on 20 paired tasks, one-sided in the stated direction.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Answer and total output tokens per task (medians over repetitions of first attempts). Each segment runs from the answer tokens (open marker) to the total output (filled marker), so its length is the reasoning; a lone filled marker means no reasoning. Panel titles give the median per-task ratio Spec/HTML/JS for answer and total tokens. Specifications are shorter in answer tokens for almost every task, but with reasoning included they are no longer consistently shorter.

Structured Output Generation

Structured output generation studies how language models produce machine-readable, schema-compliant responses without sacrificing task accuracy.

See all 65 papers

May 31, 2026

Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales

Natural-language explanations are often treated as a unified interface for understanding model behavior, but different explanation sources may support simulation in different ways. This paper compares two families of explanations for question answering models: verbalized feature attributions and self-generated rationales. We evaluate them under a shared counterfactual simulation setting, using an LLM judge as predictor and measuring whether it can better predict a model's answers to follow-up questions when given its explanation. Across multiple instruction-tuned models, we analyze how explanation source, verbalization strategy, and feature granularity affect the simulatability of explanations. Our results show that explanation format and granularity affect simulatability: attribution-based explanations and self-generated rationales differ in how much they improve counterfactual prediction, with effects that vary across models and formats.

Pingjun Hong, Benjamin Roth

Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Faculty of Philological and Cultural Studies, University of Vienna, Vienna, Austria

Feature Attribution LLM Interpretability Explainability Evaluation

Ask PDF

Apr 24, 2026

Large Language Models Decide Early and Explain Later

Large Language Models often achieve strong performance by generating long intermediate chain-of-thought reasoning. However, it remains unclear when a model's final answer is actually determined during generation. If the answer is already fixed at an intermediate stage, subsequent reasoning tokens may constitute post-decision explanation, increasing inference cost and latency without improving correctness. We study the evolution of predicted answers over reasoning steps using forced answer completion, which elicits the model's intermediate predictions at partial reasoning prefixes. Focusing on Qwen3-4B and averaging results across all datasets considered, we find that predicted answers change in only 32% of queries. Moreover, once the final answer switch occurs, the model generates an average of 760 additional reasoning tokens per query, accounting for a substantial fraction of the total reasoning budget. Motivated by these findings, we investigate early stopping strategies that halt generation once the answer has stabilized. We show that simple heuristics, including probe-based stopping, can reduce reasoning token usage by 500 tokens per query while incurring only a 2% drop in accuracy. Together, our results indicate that a large portion of chain-of-thought generation is redundant and can be reduced with minimal impact on performance.

Ayan Datta, Zhixue Zhao, Bhuvanesh Verma +3

IIIT Hyderabad · University of Sheffield, UK · Goethe University, Frankfurt am Main, Germany

Efficient Language Model Reasoning LLM Inference Acceleration Early Stopping for Language Model Reasoning LLM Reasoning Reasoning Trace Analysis

Ask PDF

Aug 22, 2026

Architecture as Capability Equalizer for Coding Agents

LLM-based coding agents generate complete software systems from high-level descriptions, yet little is known about how the format of architecture specifications affects the quality of generated code or whether this effect depends on model capability. We present a controlled experiment comparing five informationally equivalent specification formats (informal prose, Mermaid diagrams with constraints and ADRs, OpenAPI, C4/Structurizr DSL, and TypeScript interface contracts with ArchUnit-style rules) across six models from three vendor families (Anthropic Claude, OpenAI GPT, Google Gemini). Across 90 multi-turn agent trials, specification format shows a strong format x model interaction. On the strongest models (Sonnet 4.6, GPT-5), format barely matters (quality spread 0.17-0.92). On weaker models, format produces spreads of 0.83-2.42 points, with code-proximate formats (OpenAPI, TypeScript contracts) recovering most of the capability gap. Mid-tier models can consume more tokens than frontier models for worse output when they enter compilation debugging loops that stronger models avoid. Self-validation rates collapse from 100% (Sonnet) to 0% (Gemini Flash) across the capability spectrum. TypeScript contracts triple API route coverage for the weakest model (33% to 100%). Structured architecture specifications serve as a capability equalizer, with value inversely proportional to model strength and the largest returns for cost-optimized deployments.

Arquimedes Canedo

Siemens Digital Industries Software Princeton, NJ, USA

AI Coding Agents Code Generation Agentic Code Generation Code Generation Evaluation Software Architecture Design

Ask PDF

Abstract

Structured Output Generation

Explore similar work

Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales

Large Language Models Decide Early and Explain Later

Architecture as Capability Equalizer for Coding Agents