Consideration Circuits: Depth Separation and Universality Beyond a Single Softmax

Junjie Xiao; Huiwen Jia

cs.LGOct 2, 2026

Consideration Circuits: Depth Separation and Universality Beyond a Single Softmax

Authors: Junjie Xiao, Huiwen Jia

Organizations: Department of Mathematics, Peking University · Department of Industrial Engineering and Operations Research, University of California, Berkeley

Abstract

Most feature-based choice models, classical and deep, score items and apply a single softmax. We introduce consideration circuits (CC), feature-based models of multi-stage choice defined by directed acyclic graphs of multinomial logit (MNL) units. Source units assign probabilities to menu items, and internal units combine predecessor distributions using MNL weights computed from their probability-weighted feature summaries. On a three-item compromise task with fixed non-collinear features, menu-independent random-utility models (RUM), including a single MNL unit, suffer an error bounded away from zero. For CC, in contrast, we establish a sharp depth--norm separation: increasing depth from $2$ to $3$ reduces the optimal maximum taste-vector norm for error $ε$ from $Θ(\log(1/ε)/ε)$ to $Θ(\log(1/ε))$ . The depth- $2$ lower bound holds for arbitrary width and menu-independent routing biases, while a five-node depth- $3$ circuit with zero routing biases attains the logarithmic rate. More generally, we characterize two geometric conditions that are necessary and sufficient for approximating arbitrary deterministic choice tables on finite menu families. Under these conditions, depth $3$ suffices, while depth $4$ achieves optimal logarithmic norm scaling whenever the family contains a non-singleton menu. In experiments, standalone tree circuits with fewer than $600$ parameters attain the lowest mean test negative log-likelihood (NLL) among the evaluated models on four fixed-pool benchmarks and the Expedia temporal split. As output heads, CC generalize the linear MNL readout and lower mean test NLL for every tested encoder on Expedia and Trivago.

Figures & tables

Figure 1: Depth- $2$ consideration circuit versus mixture of experts. Colors indicate menu-dependent (blue) and trained, menu-independent (orange) quantities. (a) The circuit sink scores source $v$ by $w^{\top}x(v)+b_{v}$ , with menu-dependent key $x(v)$ and trained query $w$ . (b) The MoE gate scores expert $v$ by $a_{v}^{\top}g(X)+b_{v}$ , with trained key $a_{v}$ and menu-dependent query $g(X)$ .

Figure 2: The smallest consideration circuit fitted to a decoy instance. All probabilities are computed from the fitted parameters (Appendix D ); edge widths are proportional to probabilities, edges below $0.01$ omitted. (a) On the menu $\{1,2\}$ both items have a $50\%$ share. (b) The decoy $3$ shifts the sink’s attention toward source $B$ : item $2$ ’s share rises to $60\%$ , a regularity violation. (c) The mechanism (schematic): the decoy draws part of $A$ ’s attention, moving $x(A)$ across the sink’s iso-utility lines (dashed), while $x(B)$ remains close to $x_{2}$ .

Figure 3: Left: a hard structure realizing the task ( 4 ): the five-node depth- $3$ graph in the hard-max limit, all edge biases zero. In this limit every summary is itself an item, and every node reduces to a strict preference ranking: each block shows a node’s ranking (top) and the item it reports on each menu (bottom); a node reports its highest-ranked among the items nominated by its in-neighbors. The sink’s row is exactly the target. Right: a matching depth- $2$ construction for the same task (schematic). This construction exploits an $\epsilon$ -attention leak: the source summary near $x_{3}$ changes by $O_{X}(\epsilon)$ across menus. The sink’s indifference line through $x_{2}$ (dashed) separates $x_{3}$ and $x_{3^{\prime}}$ , allowing the sink to favor the summary near $x_{2}$ on $\{2,3\}$ and the summary near $x_{3}$ on $\{1,2,3\}$ . Amplifying this displacement requires an additional $1/\epsilon$ factor in norm.

	Expedia		Trivago
depth	tree	dense	tree	dense
2	$2.6101$ $(0.0008)$	$2.6100$ $(0.0011)$	$2.4540$ $(0.0004)$	$2.4539$ $(0.0005)$
3	$2.5980$ $(0.0021)$	$2.5991$ $(0.0015)$	$2.4389$ $(0.0015)$	$2.4379$ $(0.0021)$
4	$2.5973$ $(0.0022)$	$2.5983$ $(0.0017)$	$2.4357$ $(0.0020)$	$2.4334$ $(0.0019)$
5	$2.5975$ $(0.0027)$	$2.5990$ $(0.0012)$	$2.4334$ $(0.0017)$	$2.4329$ $(0.0015)$
6	$2.5988$ $(0.0014)$	$2.5991$ $(0.0010)$	$2.4308$ $(0.0019)$	$2.4323$ $(0.0016)$

Table 1: Test NLL, mean (sd) over

10

restarts, of

63

-node circuits of depth

2

–

6

: trees vs. fully-connected layered DAGs (dense); at depth

2

the two coincide. Trees share one parameter count (

503

); dense graphs pay one bias per edge (up to

1153

). The depth-

6

tree is a complete binary tree.

	featureless		featured
model	SFwork	SFshop	Swissmetro	ModeCanada
MNL	$0.8299$ $(.0002)$	$1.5887$ $(.0007)$	$0.8085$ $(.0038)$	$0.6304$ $(.0041)$
NestedLogit	$0.8321$ $(.0011)$	$1.5898$ $(.0010)$	$0.8079$ $(.0038)$	$0.6268$ $(.0010)$
LC-MNL	$0.8262$ $(.0005)$	$1.5669$ $(.0010)$	$0.7518$ $(.0083)$	$0.5929$ $(.0034)$
MMNL	$0.8291$ $(.0010)$	$1.5866$ $(.0009)$	$0.7696$ $(.0004)$	$0.6212$ $(.0011)$
Exponomial	$0.8315$ $(.0007)$	$1.5841$ $(.0014)$	$0.7850$ $(.0006)$	$0.6295$ $(.0014)$

Table 2: Test NLL, mean (sd) over

10

restarts, on four fixed-pool benchmarks; best per column in bold.

\mathrm{Tree}_{15}

\mathrm{Tree}_{63}

are complete binary trees with

15

63

nodes; on the featured pair every model carries item intercepts. LC-MNL and LowRankCDM show the best size per dataset (

K\in\{2,3,8,16,32\}

, rank

\in\{1,2,3,4\}

); full grids in Appendix C .

	classical		deep					ours
	MNL	LC-MNL	DeepMNL	RUMnet	DeepSets	DeepHalo	TCNet	$\mathrm{Tree}_{15}$	$\mathrm{Tree}_{63}$
params	8	64	4.7k	4.5k	3.7k	201k	20k/37k	119	503
Expedia	$2.6574$ $(.0008)$	$2.6153$ $(.0006)$	$3.0804$ $(.5486)$	$2.6666$ $(.0412)$	$2.9113$ $(.2282)$	$2.9592$ $(.2421)$	$2.6911$ $(.0157)$	$2.6009$ $(.0026)$	$\bm{2.5983}$ $(.0018)$
Trivago	$2.6400$ $(.0006)$	$2.4631$ $(.0006)$	$2.4746$ $(.0006)$	$2.4441$ $(.0013)$	$2.4179$ $(.0032)$	$2.4140$ $(.0027)$	$\bm{2.4101}$ $(.0033)$	$2.4417$ $(.0015)$	$2.4331$ $(.0023)$

Table 3: Test NLL, mean (sd) , on the two temporally split datasets; best per dataset in bold. MNL, LC-MNL, DeepSets, and our circuits use

10

restarts; the other deep baselines use

3

per configuration, selected by mean validation NLL. TCNet’s selected size differs per dataset.

	Expedia		Trivago
encoder	alone	$+\mathrm{Tree}_{63}$	alone	$+\mathrm{Tree}_{63}$
none	$2.6574$ $(.0008)$	$\bm{2.5983}$ $(.0018)$	$2.6400$ $(.0006)$	$2.4331$ $(.0023)$
MLP(16)	$2.6798$ $(.0398)$	$2.6469$ $(.0152)$	$2.4809$ $(.0007)$	$2.4185$ $(.0027)$
MLP(64,64)	$2.7006$ $(.0401)$	$2.6621$ $(.0181)$	$2.4751$ $(.0010)$	$2.4148$ $(.0016)$
DeepSets	$2.9113$ $(.2282)$	$2.8957$ $(.1787)$	$2.4179$ $(.0032)$	$2.4114$ $(.0118)$
DeepHalo	$2.8272$ $(.1744)$	$2.6351$ $(.0117)$	$2.4151$ $(.0027)$	$2.4133$ $(.0024)$

Table 4: Encoders with their native linear readout (alone) vs. the circuit head

\mathrm{Tree}_{63}

; test NLL, mean (sd) over

10

restarts; best per dataset in bold. On Expedia the TCNet encoder uses its one-layer configuration; encoder details in Appendix C .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

model	SFwork	SFshop	Swissmetro	ModeCanada
MNL	6	8	4	4
MNL (with intercepts)	—	—	7	8
NestedLogit	9	11	9	11
LC-MNL(2)	14	18	16	18
LC-MNL(3)	21	27	24	27
LC-MNL(8)	56	72	64	72

Appendix

Table 5: Parameter counts behind Table 2 . On the featured pair the tree counts include the per-item source biases.

	featureless		featured
model	SFwork	SFshop	Swissmetro	ModeCanada
LC-MNL(2)	$0.8275$ $(.0014)$	$1.5872$ $(.0049)$	$0.7838$ $(.0029)$	$0.6098$ $(.0023)$
LC-MNL(3)	$0.8265$ $(.0008)$	$1.5844$ $(.0058)$	$0.7628$ $(.0082)$	$0.6125$ $(.0075)$
LC-MNL(8)	$0.8263$ $(.0002)$	$1.5735$ $(.0089)$	$0.7522$ $(.0091)$	$0.6021$ $(.0057)$
LC-MNL(16)	$0.8264$ $(.0004)$	$1.5670$ $(.0005)$	$0.7518$ $(.0083)$	$0.5977$ $(.0025)$
LC-MNL(32)	$0.8262$ $(.0005)$	$1.5669$ $(.0010)$	$0.7536$ $(.0073)$	$0.5929$ $(.0034)$

Appendix

Table 6: Full size grids behind the LC-MNL and LowRankCDM rows of Table 2 (test NLL, mean (sd) over

10

restarts).

Figure 4: The encoder $\Phi$ maps the menu to transformed item representations $h_{i}$ , which are passed to a consideration-circuit output head.

encoder	linear	Mix(8)	MoE(32)	HME(63)	$\mathrm{Tree}_{15}$	$\mathrm{Tree}_{63}$
Id	$2.6574$ $(.0008)$	$2.6153$ $(.0006)$	$2.6991$ $(.0506)$	$2.7017$ $(.0693)$	$2.6009$ $(.0026)$	$2.5983$ $(.0018)$
MLP(16)	$2.6798$ $(.0398)$	$2.6786$ $(.0215)$	$2.7462$ $(.0997)$	$2.7450$ $(.1234)$	$2.6515$ $(.0349)$	$2.6469$ $(.0152)$
MLP(64,64)	$2.7006$ $(.0401)$	$2.7224$ $(.0748)$	$2.7887$ $(.0786)$	$2.8022$ $(.0585)$	$2.6607$ $(.0138)$	$2.6621$ $(.0181)$
DeepSets	$2.9113$ $(.2282)$	$2.8807$ $(.2945)$	$2.7157$ $(.0731)$	$2.8083$ $(.1816)$	$2.8132$ $(.1640)$	$2.8957$ $(.1787)$
DeepHalo	$2.8272$ $(.1744)$	$2.6854$ $(.0367)$	$2.7594$ $(.0953)$	$2.8182$ $(.1155)$	$2.6490$ $(.0337)$	$2.6351$ $(.0117)$
TCNet	$2.7377$ $(.0686)$	$2.7063$ $(.0398)$	$2.6809$ $(.0413)$	$2.6852$ $(.0342)$	$2.6834$ $(.0380)$	$2.6611$ $(.0322)$

Appendix

Table 7: Encoder

\times

head on Expedia (temporal split): test NLL, mean (sd) over

10

restarts.

encoder	linear	Mix(8)	MoE(32)	HME(63)	$\mathrm{Tree}_{15}$	$\mathrm{Tree}_{63}$
Id	$2.6400$ $(.0006)$	$2.4631$ $(.0006)$	$2.4322$ $(.0013)$	$2.4348$ $(.0015)$	$2.4417$ $(.0015)$	$2.4331$ $(.0023)$
MLP(16)	$2.4809$ $(.0007)$	$2.4491$ $(.0018)$	$2.4197$ $(.0024)$	$2.4202$ $(.0027)$	$2.4229$ $(.0032)$	$2.4185$ $(.0027)$
MLP(64,64)	$2.4751$ $(.0010)$	$2.4469$ $(.0009)$	$2.4156$ $(.0010)$	$2.4156$ $(.0016)$	$2.4194$ $(.0019)$	$2.4148$ $(.0016)$
DeepSets	$2.4179$ $(.0032)$	$2.4224$ $(.0097)$	$2.4077$ $(.0024)$	$2.4085$ $(.0021)$	$2.4119$ $(.0115)$	$2.4114$ $(.0118)$
DeepHalo	$2.4151$ $(.0027)$	$2.4273$ $(.0032)$	$2.4129$ $(.0040)$	$2.4164$ $(.0051)$	$2.4141$ $(.0024)$	$2.4133$ $(.0024)$
TCNet	$2.4093$ $(.0031)$	$2.4067$ $(.0021)$	$2.4053$ $(.0017)$	$2.4045$ $(.0030)$	$2.4035$ $(.0049)$	$2.4013$ $(.0027)$

Appendix

Table 8: Encoder

\times

head on Trivago: test NLL, mean (sd) over

10

restarts.

model	params	test NLL
MNL	8	$2.6574$ $(.0008)$
$\mathrm{Tree}_{127}$	1,015	$2.5985$ $(.0030)$
$\mathrm{Tree}_{255}$	2,039	$2.5991$ $(.0029)$
$\mathrm{Tree}_{511}$	4,087	$2.5980$ $(.0018)$
$\mathrm{Tree}_{1023}$	8,183	$2.5981$ $(.0015)$
$\mathrm{Tree}_{2047}$	16,375	$2.5976$ $(.0023)$

Appendix

Table 9: The scale ablation on the Expedia temporal axis: test NLL, mean (sd) over

10

restarts; MNL row from Table 3 . Shrunk configurations: TCNet width

8/16

, one layer, two heads; DeepHalo embed

16/32

, two heads, one round.

item	quality	price	$P(\cdot\mid\{1,2\})$	$P(\cdot\mid\{1,2,3\})$
$1$	$8.0$	$\$ 100$	$50\%$	$35\%$
$2$	$9.0$	$\$ 140$	$50\%$	$60\%$
$3$ (decoy)	$8.5$	$\$ 145$	—	$\phantom{0}5\%$

Appendix

Table 10: The decoy instance: features and target choice probabilities.

menu $\{1,2\}$
	source $A$	source $B$	model	target
$P(1)$	$0.958$	$0.001$	$50.0\%$	$50\%$
$P(2)$	$0.042$	$0.999$	$50.0\%$	$50\%$
sink attention	$0.521$	$0.479$
menu $\{1,2,3\}$
$P(1)$	$0.847$	$0.001$	$35.0\%$	$35\%$

Appendix

Table 11: The fitted model on the two menus: per-source choice distributions, sink attention, and model vs. target shares.

Explore similar work

May 8, 2026cs.CL

How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits

The circuits framework in mechanistic interpretability aims to identify sparse subgraphs of model components that are causally responsible for a behavior, typically evaluated by measuring necessity and sufficiency. But these criteria say little about whether a circuit consistently captures how a model performs a task, or if it is specific to that task. We study these two properties, consistency and specificity, across six tasks and five models, extracting circuits at the component level (attention heads and MLP blocks) and at the level of individual MLP neurons. We find that component-level circuits are highly consistent and causally important on most tasks, but they are not specific: ablating one task's circuit damages another task's performance about as much as that task's own circuit does. Neuron-level circuits, on the other hand, exhibit higher task-specificity but are far less consistent within tasks. This is explained by circuit overlap: component-level circuits share most of their components across all task pairs, related or not, while neuron-level circuits overlap only between closely related tasks. In a case study of the components shared by the task circuits of Llama-3.2-3B, we show that they consist mostly of MLP blocks, while the few attention heads within turn out to be generic attention-sink heads. Overall, our findings raise questions about the degree to which circuits can support targeted understanding of, and intervention on, model behavior.

Michael Li, Nishant Subramani

Carnegie Mellon University · Work done while at Carnegie Mellon University. · Northeastern University +1

Mechanistic Interpretability Circuits Ablation Reusable Knowledge Reuse

Ask PDF

Jul 11, 2026cs.LG

Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter

Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively. In autoregressive rollouts, the first assumption requires depth's per-step precision to survive composition. We test this with a pre-registered instrument, the shallow penalty

ρ=\mathrm{err}(\text{shallowest-exit rollout})/\mathrm{err}(\text{full-depth rollout})

, across nine DeepMind Control tasks under matched single-step (

K=1

) and multi-step (

K=4

) training, three seeds each. We find three regimes: on 6/9 tasks depth helps rollouts (intrinsic,

ρ

up to

4.7\times

), on 2/9 the shallow exits beat the full stack (inversion,

ρ

down to

0.85\times

), and one is flat. The robust inversion (cheetah) is not a property of the dynamics but is created by training: an ablation supervising early exits only at the first rollout step erases it (

ρ: 0.87\to1.18

n=8

Δ=+0.31

), while an intrinsic-tradeoff task is unaffected -- a double dissociation we call the routability catch-22, since the supervision that makes exits routable is what trains them to out-roll the full stack. The regime is partly predictable a priori: observation/action dimensionality and one-step model error correlate with

ρ

|\text{Spearman}|\approx0.75

(

n=9

). Inside a CEM planner,

ρ

's sign predicts whether planning benefits from depth, most sharply on the inversion task, where shallow planning beats deep. Finally, three cautions: a task's regime depends on the metric space, the rollout horizon, and the encoder. All thresholds and gates were fixed before the compute campaign, including a pre-registered negative for the hypothesis that motivated the study.

Achyuthan Sivasankar

New York University

Autoregressive Rollout World Model Planning Latent World Models Trajectory-Level Credit

Ask PDF

Oct 7, 2026cs.LG

Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation

Adaptive computation aims to improve language-model inference by tailoring execution to each input. For layer programs, oracle evaluations use known answers to estimate the potential gain from this flexibility, before a practical selector is available. However, a gain from selection does not by itself explain why the chosen programs help. This study examines this distinction using 32 layer-skipping and repetition programs on two models and 4,413 multiple-choice items. The analysis compares their gains over a fixed action selected without the evaluation prompt with those of input-blind perturbations at the same sites, re-evaluating selections on another prompt. With shared option order, the controls give 10.2-11.8 and 15.6-19.4 percentage points of headroom on Qwen3-4B-Base and Llama-3.1-8B, exceeding the real programs' 9.0 and 10.1 in all three random-direction draws per model. They match answer-change rate only, and the ordering depends on the menu: in post hoc comparisons, real programs lead on Llama's repeat-only menu in every draw. A smaller KL-calibrated comparison, including an input-dependent control, favours real programs in point estimate, with inconclusive corrected tests. Fixed letter offsets produce headroom of similar scale. Rotating options sharply reduces both families' headroom, while leaving positive real-minus-control differences of 1.4-2.3 and 3.7-4.5 points; their magnitudes and statistical support depend on further adjustments and the reference. A supplementary generated-answer test finds that search-selected programs keep a 26.0-point advantage over programs selected for other problems after rewording, without a placebo comparison. These results show that substantial headroom can persist across prompts with shared option order without establishing a benefit specific to the selected layer computation; neither ordering against these controls identifies that benefit.

Yibei Guo, Rui Liu

College of Aeronautics and Engineering, Kent State University

Topic pending

Ask PDF