Consideration Circuits: Depth Separation and Universality Beyond a Single Softmax

Junjie Xiao; Huiwen Jia

cs.LGOct 2, 2026

Consideration Circuits: Depth Separation and Universality Beyond a Single Softmax

Authors: Junjie Xiao, Huiwen Jia

Organizations: Department of Mathematics, Peking University · Department of Industrial Engineering and Operations Research, University of California, Berkeley

Abstract

Most feature-based choice models, classical and deep, score items and apply a single softmax. We introduce consideration circuits (CC), feature-based models of multi-stage choice defined by directed acyclic graphs of multinomial logit (MNL) units. Source units assign probabilities to menu items, and internal units combine predecessor distributions using MNL weights computed from their probability-weighted feature summaries. On a three-item compromise task with fixed non-collinear features, menu-independent random-utility models (RUM), including a single MNL unit, suffer an error bounded away from zero. For CC, in contrast, we establish a sharp depth--norm separation: increasing depth from $2$ to $3$ reduces the optimal maximum taste-vector norm for error $ε$ from $Θ(\log(1/ε)/ε)$ to $Θ(\log(1/ε))$ . The depth- $2$ lower bound holds for arbitrary width and menu-independent routing biases, while a five-node depth- $3$ circuit with zero routing biases attains the logarithmic rate. More generally, we characterize two geometric conditions that are necessary and sufficient for approximating arbitrary deterministic choice tables on finite menu families. Under these conditions, depth $3$ suffices, while depth $4$ achieves optimal logarithmic norm scaling whenever the family contains a non-singleton menu. In experiments, standalone tree circuits with fewer than $600$ parameters attain the lowest mean test negative log-likelihood (NLL) among the evaluated models on four fixed-pool benchmarks and the Expedia temporal split. As output heads, CC generalize the linear MNL readout and lower mean test NLL for every tested encoder on Expedia and Trivago.

Figures & tables

Figure 1: Depth- $2$ consideration circuit versus mixture of experts. Colors indicate menu-dependent (blue) and trained, menu-independent (orange) quantities. (a) The circuit sink scores source $v$ by $w^{\top}x(v)+b_{v}$ , with menu-dependent key $x(v)$ and trained query $w$ . (b) The MoE gate scores expert $v$ by $a_{v}^{\top}g(X)+b_{v}$ , with trained key $a_{v}$ and menu-dependent query $g(X)$ .

Figure 2: The smallest consideration circuit fitted to a decoy instance. All probabilities are computed from the fitted parameters (Appendix D ); edge widths are proportional to probabilities, edges below $0.01$ omitted. (a) On the menu $\{1,2\}$ both items have a $50\%$ share. (b) The decoy $3$ shifts the sink’s attention toward source $B$ : item $2$ ’s share rises to $60\%$ , a regularity violation. (c) The mechanism (schematic): the decoy draws part of $A$ ’s attention, moving $x(A)$ across the sink’s iso-utility lines (dashed), while $x(B)$ remains close to $x_{2}$ .

Figure 3: Left: a hard structure realizing the task ( 4 ): the five-node depth- $3$ graph in the hard-max limit, all edge biases zero. In this limit every summary is itself an item, and every node reduces to a strict preference ranking: each block shows a node’s ranking (top) and the item it reports on each menu (bottom); a node reports its highest-ranked among the items nominated by its in-neighbors. The sink’s row is exactly the target. Right: a matching depth- $2$ construction for the same task (schematic). This construction exploits an $\epsilon$ -attention leak: the source summary near $x_{3}$ changes by $O_{X}(\epsilon)$ across menus. The sink’s indifference line through $x_{2}$ (dashed) separates $x_{3}$ and $x_{3^{\prime}}$ , allowing the sink to favor the summary near $x_{2}$ on $\{2,3\}$ and the summary near $x_{3}$ on $\{1,2,3\}$ . Amplifying this displacement requires an additional $1/\epsilon$ factor in norm.

	Expedia		Trivago
depth	tree	dense	tree	dense
2	$2.6101$ $(0.0008)$	$2.6100$ $(0.0011)$	$2.4540$ $(0.0004)$	$2.4539$ $(0.0005)$
3	$2.5980$ $(0.0021)$	$2.5991$ $(0.0015)$	$2.4389$ $(0.0015)$	$2.4379$ $(0.0021)$
4	$2.5973$ $(0.0022)$	$2.5983$ $(0.0017)$	$2.4357$ $(0.0020)$	$2.4334$ $(0.0019)$
5	$2.5975$ $(0.0027)$	$2.5990$ $(0.0012)$	$2.4334$ $(0.0017)$	$2.4329$ $(0.0015)$
6	$2.5988$ $(0.0014)$	$2.5991$ $(0.0010)$	$2.4308$ $(0.0019)$	$2.4323$ $(0.0016)$

Table 1: Test NLL, mean (sd) over

10

restarts, of

63

-node circuits of depth

2

–

6

: trees vs. fully-connected layered DAGs (dense); at depth

2

the two coincide. Trees share one parameter count (

503

); dense graphs pay one bias per edge (up to

1153

). The depth-

6

tree is a complete binary tree.

	featureless		featured
model	SFwork	SFshop	Swissmetro	ModeCanada
MNL	$0.8299$ $(.0002)$	$1.5887$ $(.0007)$	$0.8085$ $(.0038)$	$0.6304$ $(.0041)$
NestedLogit	$0.8321$ $(.0011)$	$1.5898$ $(.0010)$	$0.8079$ $(.0038)$	$0.6268$ $(.0010)$
LC-MNL	$0.8262$ $(.0005)$	$1.5669$ $(.0010)$	$0.7518$ $(.0083)$	$0.5929$ $(.0034)$
MMNL	$0.8291$ $(.0010)$	$1.5866$ $(.0009)$	$0.7696$ $(.0004)$	$0.6212$ $(.0011)$
Exponomial	$0.8315$ $(.0007)$	$1.5841$ $(.0014)$	$0.7850$ $(.0006)$	$0.6295$ $(.0014)$

Table 2: Test NLL, mean (sd) over

10

restarts, on four fixed-pool benchmarks; best per column in bold.

\mathrm{Tree}_{15}

\mathrm{Tree}_{63}

are complete binary trees with

15

63

nodes; on the featured pair every model carries item intercepts. LC-MNL and LowRankCDM show the best size per dataset (

K\in\{2,3,8,16,32\}

, rank

\in\{1,2,3,4\}

); full grids in Appendix C .

	classical		deep					ours
	MNL	LC-MNL	DeepMNL	RUMnet	DeepSets	DeepHalo	TCNet	$\mathrm{Tree}_{15}$	$\mathrm{Tree}_{63}$
params	8	64	4.7k	4.5k	3.7k	201k	20k/37k	119	503
Expedia	$2.6574$ $(.0008)$	$2.6153$ $(.0006)$	$3.0804$ $(.5486)$	$2.6666$ $(.0412)$	$2.9113$ $(.2282)$	$2.9592$ $(.2421)$	$2.6911$ $(.0157)$	$2.6009$ $(.0026)$	$\bm{2.5983}$ $(.0018)$
Trivago	$2.6400$ $(.0006)$	$2.4631$ $(.0006)$	$2.4746$ $(.0006)$	$2.4441$ $(.0013)$	$2.4179$ $(.0032)$	$2.4140$ $(.0027)$	$\bm{2.4101}$ $(.0033)$	$2.4417$ $(.0015)$	$2.4331$ $(.0023)$

Table 3: Test NLL, mean (sd) , on the two temporally split datasets; best per dataset in bold. MNL, LC-MNL, DeepSets, and our circuits use

10

restarts; the other deep baselines use

3

per configuration, selected by mean validation NLL. TCNet’s selected size differs per dataset.

	Expedia		Trivago
encoder	alone	$+\mathrm{Tree}_{63}$	alone	$+\mathrm{Tree}_{63}$
none	$2.6574$ $(.0008)$	$\bm{2.5983}$ $(.0018)$	$2.6400$ $(.0006)$	$2.4331$ $(.0023)$
MLP(16)	$2.6798$ $(.0398)$	$2.6469$ $(.0152)$	$2.4809$ $(.0007)$	$2.4185$ $(.0027)$
MLP(64,64)	$2.7006$ $(.0401)$	$2.6621$ $(.0181)$	$2.4751$ $(.0010)$	$2.4148$ $(.0016)$
DeepSets	$2.9113$ $(.2282)$	$2.8957$ $(.1787)$	$2.4179$ $(.0032)$	$2.4114$ $(.0118)$
DeepHalo	$2.8272$ $(.1744)$	$2.6351$ $(.0117)$	$2.4151$ $(.0027)$	$2.4133$ $(.0024)$

Table 4: Encoders with their native linear readout (alone) vs. the circuit head

\mathrm{Tree}_{63}

; test NLL, mean (sd) over

10

restarts; best per dataset in bold. On Expedia the TCNet encoder uses its one-layer configuration; encoder details in Appendix C .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

model	SFwork	SFshop	Swissmetro	ModeCanada
MNL	6	8	4	4
MNL (with intercepts)	—	—	7	8
NestedLogit	9	11	9	11
LC-MNL(2)	14	18	16	18
LC-MNL(3)	21	27	24	27
LC-MNL(8)	56	72	64	72

Appendix

Table 5: Parameter counts behind Table 2 . On the featured pair the tree counts include the per-item source biases.

	featureless		featured
model	SFwork	SFshop	Swissmetro	ModeCanada
LC-MNL(2)	$0.8275$ $(.0014)$	$1.5872$ $(.0049)$	$0.7838$ $(.0029)$	$0.6098$ $(.0023)$
LC-MNL(3)	$0.8265$ $(.0008)$	$1.5844$ $(.0058)$	$0.7628$ $(.0082)$	$0.6125$ $(.0075)$
LC-MNL(8)	$0.8263$ $(.0002)$	$1.5735$ $(.0089)$	$0.7522$ $(.0091)$	$0.6021$ $(.0057)$
LC-MNL(16)	$0.8264$ $(.0004)$	$1.5670$ $(.0005)$	$0.7518$ $(.0083)$	$0.5977$ $(.0025)$
LC-MNL(32)	$0.8262$ $(.0005)$	$1.5669$ $(.0010)$	$0.7536$ $(.0073)$	$0.5929$ $(.0034)$

Appendix

Table 6: Full size grids behind the LC-MNL and LowRankCDM rows of Table 2 (test NLL, mean (sd) over

10

restarts).

Figure 4: The encoder $\Phi$ maps the menu to transformed item representations $h_{i}$ , which are passed to a consideration-circuit output head.

encoder	linear	Mix(8)	MoE(32)	HME(63)	$\mathrm{Tree}_{15}$	$\mathrm{Tree}_{63}$
Id	$2.6574$ $(.0008)$	$2.6153$ $(.0006)$	$2.6991$ $(.0506)$	$2.7017$ $(.0693)$	$2.6009$ $(.0026)$	$2.5983$ $(.0018)$
MLP(16)	$2.6798$ $(.0398)$	$2.6786$ $(.0215)$	$2.7462$ $(.0997)$	$2.7450$ $(.1234)$	$2.6515$ $(.0349)$	$2.6469$ $(.0152)$
MLP(64,64)	$2.7006$ $(.0401)$	$2.7224$ $(.0748)$	$2.7887$ $(.0786)$	$2.8022$ $(.0585)$	$2.6607$ $(.0138)$	$2.6621$ $(.0181)$
DeepSets	$2.9113$ $(.2282)$	$2.8807$ $(.2945)$	$2.7157$ $(.0731)$	$2.8083$ $(.1816)$	$2.8132$ $(.1640)$	$2.8957$ $(.1787)$
DeepHalo	$2.8272$ $(.1744)$	$2.6854$ $(.0367)$	$2.7594$ $(.0953)$	$2.8182$ $(.1155)$	$2.6490$ $(.0337)$	$2.6351$ $(.0117)$
TCNet	$2.7377$ $(.0686)$	$2.7063$ $(.0398)$	$2.6809$ $(.0413)$	$2.6852$ $(.0342)$	$2.6834$ $(.0380)$	$2.6611$ $(.0322)$

Appendix

Table 7: Encoder

\times

head on Expedia (temporal split): test NLL, mean (sd) over

10

restarts.

encoder	linear	Mix(8)	MoE(32)	HME(63)	$\mathrm{Tree}_{15}$	$\mathrm{Tree}_{63}$
Id	$2.6400$ $(.0006)$	$2.4631$ $(.0006)$	$2.4322$ $(.0013)$	$2.4348$ $(.0015)$	$2.4417$ $(.0015)$	$2.4331$ $(.0023)$
MLP(16)	$2.4809$ $(.0007)$	$2.4491$ $(.0018)$	$2.4197$ $(.0024)$	$2.4202$ $(.0027)$	$2.4229$ $(.0032)$	$2.4185$ $(.0027)$
MLP(64,64)	$2.4751$ $(.0010)$	$2.4469$ $(.0009)$	$2.4156$ $(.0010)$	$2.4156$ $(.0016)$	$2.4194$ $(.0019)$	$2.4148$ $(.0016)$
DeepSets	$2.4179$ $(.0032)$	$2.4224$ $(.0097)$	$2.4077$ $(.0024)$	$2.4085$ $(.0021)$	$2.4119$ $(.0115)$	$2.4114$ $(.0118)$
DeepHalo	$2.4151$ $(.0027)$	$2.4273$ $(.0032)$	$2.4129$ $(.0040)$	$2.4164$ $(.0051)$	$2.4141$ $(.0024)$	$2.4133$ $(.0024)$
TCNet	$2.4093$ $(.0031)$	$2.4067$ $(.0021)$	$2.4053$ $(.0017)$	$2.4045$ $(.0030)$	$2.4035$ $(.0049)$	$2.4013$ $(.0027)$

Appendix

Table 8: Encoder

\times

head on Trivago: test NLL, mean (sd) over

10

restarts.

model	params	test NLL
MNL	8	$2.6574$ $(.0008)$
$\mathrm{Tree}_{127}$	1,015	$2.5985$ $(.0030)$
$\mathrm{Tree}_{255}$	2,039	$2.5991$ $(.0029)$
$\mathrm{Tree}_{511}$	4,087	$2.5980$ $(.0018)$
$\mathrm{Tree}_{1023}$	8,183	$2.5981$ $(.0015)$
$\mathrm{Tree}_{2047}$	16,375	$2.5976$ $(.0023)$

Appendix

Table 9: The scale ablation on the Expedia temporal axis: test NLL, mean (sd) over

10

restarts; MNL row from Table 3 . Shrunk configurations: TCNet width

8/16

, one layer, two heads; DeepHalo embed

16/32

, two heads, one round.

item	quality	price	$P(\cdot\mid\{1,2\})$	$P(\cdot\mid\{1,2,3\})$
$1$	$8.0$	$\$ 100$	$50\%$	$35\%$
$2$	$9.0$	$\$ 140$	$50\%$	$60\%$
$3$ (decoy)	$8.5$	$\$ 145$	—	$\phantom{0}5\%$

Appendix

Table 10: The decoy instance: features and target choice probabilities.

menu $\{1,2\}$
	source $A$	source $B$	model	target
$P(1)$	$0.958$	$0.001$	$50.0\%$	$50\%$
$P(2)$	$0.042$	$0.999$	$50.0\%$	$50\%$
sink attention	$0.521$	$0.479$
menu $\{1,2,3\}$
$P(1)$	$0.847$	$0.001$	$35.0\%$	$35\%$

Appendix

Table 11: The fitted model on the two menus: per-source choice distributions, sink attention, and model vs. target shares.

Explore similar work

How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits
May 8, 2026Michael Li, Nishant SubramaniMechanistic Interpretability Circuits
Carnegie Mellon University · Work done while at Carnegie Mellon University. · Northeastern University +1
Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter
Jul 11, 2026Achyuthan SivasankarAutoregressive Rollout World Model Planning
New York University
Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation
Oct 7, 2026Yibei Guo, Rui Liu
College of Aeronautics and Engineering, Kent State University

May 8, 2026, cs.CLJ/K move · Enter open · S save

How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits

Michael Li, Nishant Subramani

Carnegie Mellon University · Work done while at Carnegie Mellon University. · Northeastern University+1

Mechanistic Interpretability Circuits Ablation Reusable Knowledge Reuse

Open paper AskPDF arXiv

The circuits framework in mechanistic interpretability aims to identify sparse subgraphs of model components that are causally responsible for a behavior, typically evaluated by measuring necessity and sufficiency. But these criteria say little about whether a circuit consistently captures how a model performs a task, or if it is specific to that task. We study these two properties, consistency and specificity, across six tasks and five models, extracting circuits at the component level (attention heads and MLP blocks) and at the level of individual MLP neurons. We find that component-level circuits are highly consistent and causally important on most tasks, but they are not specific: ablating one task's circuit damages another task's performance about as much as that task's own circuit does. Neuron-level circuits, on the other hand, exhibit higher task-specificity but are far less consistent within tasks. This is explained by circuit overlap: component-level circuits share most of their components across all task pairs, related or not, while neuron-level circuits overlap only between closely related tasks. In a case study of the components shared by the task circuits of Llama-3.2-3B, we show that they consist mostly of MLP blocks, while the few attention heads within turn out to be generic attention-sink heads. Overall, our findings raise questions about the degree to which circuits can support targeted understanding of, and intervention on, model behavior.

Abstract

Explore similar work

How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits

Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter

Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation