Large language models (LLMs) have demonstrated strong performance on structured reasoning tasks, but what they encode and whether it informs model behavior remain unclear. We investigate this question through geometric reasoning, using parametric CAD constraints as a controlled testbed for separating local pairwise relations from sketch-level constraint status. By probing the hidden states of six frozen decoder-only LLMs, we examine four properties: linear decodability, forced-choice generation, activation-level influence, and behavioral steerability. Pretraining substantially improves the decoding of local geometric relations, and this advantage persists after accounting for positional cues with shuffled-order controls. In contrast, sketch-level DOF status is already highly decodable from randomly initialized representations and improves only modestly with pretraining, indicating that much of its probe performance is available without learned weights. Further analyses show that decodable information is not always actionable. Generation often fails to express this information, and on the two intervention-tested backbones, activation-restoration effects at the patched entity position vanish while decodability persists across depth. Mean-difference steering also does not reliably control outputs. These results show that decodability, generation, activation-level influence, and steerability can diverge in the tested setting. The audit provides a controlled way to distinguish failures to encode geometric structure from failures to express or control encoded information.
Figures & tables
Figure 1: Overview of the four-part audit. Geometry-only sketches are serialized and passed through a frozen LLM, whose representations are evaluated for linear decodability, forced-choice generation, sensitivity to activation restoration, and steering-based control.
Figure 2: P1 macro-F1 across relative depth for all six models and controls. Orange brackets show the drop under entity-order shuffling; green marks show the remaining gain over random initialization.
Peak F1
Selectivity
Dissociation
Model
P1
P2 †
P1
P2 †
P1 layer/total
DI at ℓ∗ [95% CI]
Net gain ‡
Baselines
Chance
.125
.333
–
–
–
–
–
Pure input
.359
.380
–
–
–
–
–
Random-init (no pretraining)
Qwen2.5-0.5B
.598
.679
.468
.349
3/24
–
–
Table 1: Probe performance across models. P1 columns and P1 layer/total report each checkpoint’s own P1 peak. † P2 is evaluated at the corresponding pretrained model’s P1 peak, ℓ∗ . DI evaluates all four trained and random-init terms at this pretrained ℓ∗ (Eq. 2 ). Net gain ‡ is shuffled-order pretrained P1 macro-F1 minus the random-init P1 peak.
Model
P1 probe F1
P3 gen. F1
Gap
Qwen2.5-0.5B
0.725
0.097
0.628
Qwen2.5-1.5B
0.724
0.072
0.652
Qwen2.5-3B
0.714
0.081
0.633
Qwen2.5-7B
0.719
0.259
0.460
Mistral-7B
0.725
0.025
0.700
Llama-3.1-8B
0.734
0.151
0.583
Table 2: Matched P1 probe and P3 generation macro-F1 on the same held-out pairs and eight-class label set (chance =0.125 ).
Figure 3: P3 predicted-class distributions across six models. Boxes mark the dominant class, and dashes denote exact zeros. Per-class accuracies are reported in Appendix C.1 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Worked example of the geometry-only protocol. A source sketch (a) is serialized with all EdgeOp labels excluded (b) and used to construct the matched P1–P3 evaluation targets (c).
Figure 5: P1 pairwise constraint classes and P2 structural DOF labels in SketchGraphs. Highlighted entities indicate the pair probed in P1; EdgeOp tokens are excluded from the LLM input.
Model (random init)
Peak P1
Peak P2
P1 layer
Qwen2.5-0.5B
0.598
0.681
3
Qwen2.5-1.5B
0.567
0.686
4
Qwen2.5-3B
0.560
0.695
5
Qwen2.5-7B
0.560
0.691
4
Llama-3.1-8B
0.549
0.684
14
Mistral-7B
0.563
0.692
5
Appendix
Table 3: Task-specific peak macro-F1 scores for the random-init controls. The final column reports the layer of the P1 peak.
Model
ℓ∗
Raw DI
DInorm
Qwen2.5-0.5B
10
.106
.107
Qwen2.5-1.5B
12
.141
.143
Qwen2.5-3B
21
.167
.178
Qwen2.5-7B
20
.152
.157
Llama-3.1-8B
14
.142
.147
Mistral-7B
14
.125
.126
Appendix
Table 4: Raw and chance-normalized DI at each model’s P1-selected layer ℓ∗ .
Figure 6: P3 accuracy by model and true constraint class. Dashes denote exact zeros.
Model
Coin (222)
Par (239)
Perp (247)
Tan (256)
Eq (261)
Mid (249)
Con (264)
NoC (262)
Qwen2.5-0.5B
0.21/0.27
0.10/0.15
0.04/0.07
0.65/0.96
–/–
–/–
–/–
–/–
Qwen2.5-1.5B
0.01/0.00
–/–
0.22/0.40
0.77/0.95
–/–
–/–
–/–
0.00/–
Qwen2.5-3B
0.35/0.36
–/–
0.02/0.07
0.63/0.96
0.00/0.00
–/–
0.00/0.01
–/–
Qwen2.5-7B
0.08/0.11
0.36/0.79
0.09/0.36
0.34/0.96
0.00/–
0.00/0.01
0.12/0.53
0.00/0.02
Mistral-7B
1.00/1.00
0.00/–
–/–
–/–
–/–
–/–
–/–
0.00/–
Llama-3.1-8B
0.28/0.23
0.02/0.05
0.01/0.01
0.10/0.38
0.01/0.01
–/–
0.03/0.14
0.56/0.81
Appendix
Table 5: P3 predicted-class frequency and per-class accuracy. Each cell reports the fraction of all predictions assigned to that class, followed by accuracy among examples with that true label. Column headers give the true-class counts ( n=2,000 total). Bold marks each model’s most frequently predicted class. Values are rounded to two decimals; dashes denote exact zeros.
Class
Mistral-7B
Qwen2.5-7B
Coincident
0.345
0.129
Parallel
0.153
0.250
Perpendicular
0.090
0.152
Tangent
0.043
0.212
Equal
0.075
0.032
Midpoint
0.028
0.038
Appendix
Table 6: Mean P3 class probabilities across five content-free templates. Bold marks the highest-probability class for each model.
Figure 7: Layerwise P1 restoration rates and decodability for Qwen2.5-3B and Llama-3.1-8B. Annotations mark the strongest nontrivial restoration layer and each model’s P1 decodability peak. Shaded bands show 95% bootstrap confidence intervals.
Qwen2.5-3B
Llama-3.1-8B
α
Mean diff.
Random
Mean diff.
Random
0.5
0.0%
0.05–0.2%
2.0–2.5%
2.75–3.0%
1.0
0.0–1.0%
0.15–0.4%
5.0–5.5%
5.7–5.75%
2.0
1.0%
0.4–0.5%
9.0–9.5%
10.8–10.9%
4.0
2.5%
0.75–0.9%
12.5%
21.9–22.0%
8.0
11.5–12.0%
2.85–2.9%
16.0–16.5%
25.0–25.2%
Appendix
Table 7: P1 label-change rates at layer 4 under mean-difference and matched random-direction interventions. Cells report ranges across two independent runs.
Model layer
Macro-F1
Selectivity
0
0.484
0.354
5
0.567
0.439
10
0.601
0.475
15
0.612
0.489
21
0.640
0.518
26
0.643
0.515
Appendix
Table 8: P1 macro-F1 and selectivity across sampled Qwen2.5-3B layers on Fusion 360 Gallery.
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.