Variable roles describe recurring behaviors such as accumulation, traversal, and retaining the best candidate encountered. We use Sajaniemi's independently defined taxonomy to study which role distinctions can be recovered from large language models, how sparse autoencoders (SAEs) retain those distinctions, and whether associated directions support control of generation. We introduce static annotation rules for all eleven roles and a corpus of 126,926 natural-code files across five languages. Across five model-SAE systems, residual probes outperform individual SAE features and tested lexical baselines, with modest losses after neutral target renaming. Combining SAE features closes much of the single-feature gap, but probes on SAE reconstructions are less sensitive to role-changing edits reviewed by LLMs. High paired ranking accuracy coexists with frequent false positives at fixed thresholds. During generation, interventions often change role annotations at the cost of correctness; changes accompanied by passing tests show no clear aggregate advantage over random controls. The benchmark separates role-related recoverability, SAE fidelity, and correctness-preserving control through matched prediction, source edits, and generation interventions.
Figures & tables
Figure 1: Variable roles connect code annotation, representation analysis, and steering. (1) Code Property Graph (CPG) queries annotate all eleven Sajaniemi variable roles at scale. (2) Probes and SAE features recover role information; bars show matched-pair test average precision (AP) for Gemma 3 12B (four-layer mean and range). Code and graph sketches illustrate the method. (3) In Gemma 3 4B, steering a most-wanted-holder (MWH) feature changes a stub into a working search.
Figure 2: Overview of conceptual variable roles according to Sajaniemi (2002) and Sajaniemi and Navarro Prieto (2005) . Matching colors and symbols link definitions to variable occurrences in Python code. Examples are deliberately simple, chosen for illustration. Appendix A shows more complex examples from real-world code.
Work
Signal
Languages
Roles
Corpus
Labeled data
Gerdt and Sajaniemi (2004)
Static analysis
Pascal
3
NR
NR
Bishop and Johnson (2005)
Static analysis
Java
Checking
17 programs
NR
Gerdt and Sajaniemi (2006)
Static + learning
Pascal
9
87 programs
305 vars.
Demyanova et al. (2013)
Static analysis
C
14 custom
5.2 KLOC †
NR
Lui et al. (2020)
Traces + NN
C
4 classes
110 programs
8,907 cases
Hussain et al. (2023)
Syntax rules
Java
2
14,022,318 methods
1,438,035 vars.
Table 1: Automatic variable-role analysis and reported corpus sizes. All but Demyanova et al. use Sajaniemi’s taxonomy. NR: not reported; vars.: role-labeled variables; NN: neural classifier. † : role-definition sample. Appendix B details annotation units, breakdowns, and comparability.
Figure 3: Model representations support role recovery beyond the tested surface baselines. Residual probes attain higher mean test average precision (AP) than both surface baselines and individual SAE features for every role and model. Evaluation uses matched C, C++, JavaScript and Python pairs from one source-group split. Model bars average AP over four prescribed layers; whiskers show the layer range. Gray baselines use character n -grams of names (N) or code prefixes through the causal readout (C). Role colors and symbols follow Figure 2 ; solid and hatched bars denote Probe and validation-selected SAE feature.
Predictor
Matched AUROC
Counterfactual paired accuracy
Residual probe
0.906–0.913
0.813–0.877
SAE reconstruction probe
0.857–0.881
0.739–0.784
Top-64 SAE probe
0.853–0.878
0.722–0.753
Single SAE feature
0.722–0.782
0.622–0.713
Table 2: Multiple-feature readouts close much of the single-feature gap; reconstruction probes have lower counterfactual sensitivity. Values span five model means, each weighting eleven roles and four prescribed layers equally, with matched-negative training. Ranges show variation across systems; paired uncertainty for reconstruction-minus-residual differences is reported in the text. Counterfactual accuracy compares each original only with its corresponding source-reviewed edit.
Figure 4: Role-associated latents connect code updates to broader extrema, Boolean, and indexing concepts. Role colors and symbols follow Figure 2 ; descriptions and prose come from Neuronpedia, and code from our training corpus. Highlight intensity indicates latent activation strength on a linear scale within each latent; return marks denote newlines. Code activations use standalone snippets; prose windows retain their original Neuronpedia contexts.
Figure 5: Positive steering can repair stubs but can also break correct programs. Each pair shows held-out MBPP excerpts; test counts are passing supplied assertions out of three, and role bits denote extractor output. G2/G3 denote Gemma 2 2B/Gemma 3 4B; MWH denotes most-wanted holder. The selected repairs and failures are illustrative, while the random cases come from a uniform draw without replacement. Outcomes evaluate complete outputs, including any omitted foreign-language tails.
Figure 6: Causal effects of representative role-associated SAE features in Gemma 2 2B. (a) Scores for one selected feature per role and same-layer random SAE controls (hatched means; range whiskers); n/a denotes undefined necessity. (b) Causal scores versus changes in MBPP Pass@1; shapes match role labels in (a). Removal suggests causal involvement with limited specificity and substantial correctness costs; positive steering provides weak evidence of sufficiency.
Gemma 2 2B
Gemma 3 4B
Role
Representative syntax
Pos. steer.
Orth.
Pos. steer.
Orth.
Gatherer
Augmented assignment
+1.9 (107)
−58.8 (17)
+2.3 (44)
−50.0 (4)
Most-wanted holder
if statement
−4.3 (92)
−35.0 (20)
+17.6 (17)
−75.0 (4)
One-way flag
Boolean literal
−5.9 (17)
–
−11.1 (18)
–
Stepper
range call
+20.3 (59)
−20.0 (40)
+9.3 (54)
−63.6 (11)
Walker
Subscript expression
−2.5 (40)
+0.0 (8)
−5.1 (39)
+0.0 (10)
Table 3: Role-adjacent syntax changes after intervention. Cells give intervention-minus-baseline changes (percentage points) in program frequencies of Tree-sitter patterns, with counts of pairs with two clean parses in parentheses. Patterns are structural correlates, not role definitions or evidence of full-role sufficiency. Cohorts and baselines differ by intervention; dashes mark unavailable removal generations. Counts repeat tasks across features; Appendix Q reports parse survival.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Study / coverage
Corpus and labeled examples
Scope and qualifications
Gerdt and Sajaniemi (2004) Pascal; 3 roles
Prototype program and variable counts: NR.
Fixed Value, Stepper, and Most-Recent Holder, selected using an earlier textbook survey.
Bishop and Johnson (2005) Java; role checking
17 canonical teaching programs; variable count: NR.
Checks supplied role assignments in procedural Java.
Automatic syntax rules; pool includes methods without either role. Evaluated on method-name prediction.
Appendix
Table 4: Corpus sizes and annotation scope for Table 1 . Totals combine reported splits where available. NR: not reported, not zero. Units and annotation quality are not directly comparable.
Distribution
Performance
Role
Extractor
Human
Precision
Recall
F1
Container
30
27
0.733
0.815
0.772
Fixed value
106
145
0.830
0.607
0.701
Follower
20
22
0.900
0.818
0.857
Gatherer
20
25
0.900
0.720
0.800
Most-recent holder
21
30
0.667
0.467
0.549
Appendix
Table 5: Human evaluation of the variable role extractor on 660 stratified samples. Distribution columns report the number of extractor predictions and human annotations per role. Precision, recall, and F1 are computed one-vs-rest against human labels, excluding “no role”. Scores characterize the selected subset rather than corpus-wide performance.
Partition
Files
Variables
Role pairs
Train
76,155
2,116,300
1,048,604
Validation
25,386
705,213
349,496
Test
25,385
704,017
349,527
All
126,926
3,525,530
1,747,627
Appendix
Table 6: Consolidated corpus coverage by partition and language. Role pairs count variable–role memberships; with role is the fraction of variables with at least one extracted role.
Role
Train
Valid.
Test
Total
%
Test files
Fixed value
780,012
259,968
260,007
1,299,987
36.87
24,387
Stepper
22,201
7,403
7,402
37,006
1.05
4,079
Gatherer
2,877
959
958
4,794
0.14
677
Most-recent holder
6,500
2,166
2,168
10,834
0.31
1,300
Most-wanted holder
901
298
300
1,499
0.04
217
One-way flag
4,321
1,439
1,437
7,197
0.20
1,132
Appendix
Table 7: Role support in the consolidated corpus. Percentages use all 3,525,530 variables; test files counts files in each row. Extracted (total) sums variable–role and file–role memberships across rows, not unique variables or files. “No extracted role” is not a prediction target.
Predictor
Macro AP
Macro F1
Binary majority
0.045
0.000
Most frequent role
0.045
0.049
Always positive
0.045
0.071
Random (uniform)
0.0451±0.0001
0.0598±0.0001
Random (prior-matched)
0.0451±0.0001
0.0451±0.0004
Language + kind
0.100 [0.098, 0.101]
0.112 [0.109, 0.114]
Appendix
Table 8: Macro surface-baseline performance over eleven roles. Brackets: 95% intervals from 500 source-group bootstrap resamples, conditional on fitted models; ± : standard deviation over 500 random draws.
Role
Prior AP
Name AP
Code AP
Combined AP
Combined F1
Fixed value
0.3693
0.594
0.785
0.822
0.752
Stepper
0.0105
0.363
0.649
0.679
0.690
Gatherer
0.0014
0.075
0.423
0.479
0.535
Most-recent holder
0.0031
0.011
0.222
0.223
0.317
Most-wanted holder
0.0004
0.056
0.388
0.458
0.488
One-way flag
0.0020
0.052
0.516
0.665
0.658
Appendix
Table 9: Per-role surface predictability on the consolidated corpus. Prior AP is test prevalence; code denotes local code, and combined adds names and boolean properties.
Model
SAE release
SAE ID
Layers
Gemma 2 2B
gemma-scope-2b-pt-res-canonical
layer_{l}/width_16k/canonical
6, 13, 17, 22
Gemma 3 4B
gemma-scope-2-4b-pt-res
layer_{l}_width_65k_l0_medium
9, 17, 22, 29
Gemma 3 12B
gemma-scope-2-12b-pt-res
layer_{l}_width_65k_l0_medium
12, 24, 31, 41
Llama 3.1 8B
llama_scope_lxr_32x
l{l}r_32x
8, 16, 21, 27
Qwen3.5 9B
qwen-scope-3.5-9b-base-w64k-l100
layer{l}
8, 16, 21, 27
Appendix
Table 10: Sparse autoencoder configurations used for each model.
Figure 7: Residual probes recover more role information than individual SAE features, with modest sensitivity to neutral target names. (a) Matched-negative Test AUROC for all eleven roles; points average five models and their four prescribed layers, and lines span the five model means. (b) Change in AUROC after renaming both members of 100 positive/negative pairs per role, with fits and thresholds fixed. Intervals are paired 95% source-group bootstrap intervals; negative values indicate lower discrimination after renaming.
Figure 9: Language-conditioned discrimination and implementation recall differ across roles and predictors. (a–b) AUROC averaged over five models and four prescribed layers, targeting 50 matched pairs per role/language. Gray cells show cohorts below 20 pairs or ten positive source groups; a dash marks an unsupported cohort. (c) Recall on metadata-defined positive subsets, with sample counts and 95% source-group bootstrap intervals. Circles denote residual probes and diamonds single SAE features; MWH and MRH abbreviate most-wanted and most-recent holder. Implicit walkers are all Python, whereas 49 of 50 structural walkers are C/C++, so their pooled contrast also changes language.
Comparison
Population and split
Selection and outcome
Corpus baselines
3,525,530 variables; three training folds, one validation fold, one Test fold
Automatic annotations; metadata and lexical prediction on the full consolidated corpus.
Representation fits
5,321 training and 9,547 validation pairs; source groups separated from Test
Training fixes normalization and feature ranking; validation chooses regularization, single feature, and threshold.
Matched Test
4,238 positive/negative pairs after exclusions; four languages
Shared examples across predictors; AP and AUROC measure rankings against automatic labels.
Counterfactual Test
1,698 accepted pairs from 948 source groups
Static source review of original and edited behavior; paired ranking and frozen-threshold decisions.
Neutral-name Test
100 positive/negative pairs per role
Both bindings renamed, with fits fixed; paired change in AUROC and binary decisions.
Feature-scale steering
Union of 121 MBPP tasks; role-specific cohorts
Eleven SAE targets each for Gemma 2B/4B and nine for 12B; ten same-layer random features per target.
Appendix
Table 11: Evaluation populations and fitted choices. Pair and target–task counts can reuse bindings or tasks; uncertainty resamples the corresponding source groups or unique tasks. The separate Gemma 12B stepper pilot uses 48 calibration and 144 Test tasks (Appendix Q ).
Figure 10: Demonstration gains differ across checkpoints. Top and middle: role-macro accuracy and AP on identical development pairs; gray lines mark 50%. Bottom: eight-minus-zero accuracy gains with paired 95% source-group bootstrap intervals (2,000 draws), conditional on the cohort, pair weights, and one demonstration set. Intervals are exploratory and unadjusted for multiple comparisons. Model labels specify version and nominal size within each family.
Figure 11: Role accuracy varies within these aggregate trends. Each checkpoint has four columns for 0/2/4/8 demonstrations: all 528 combinations, with original pair weights and a shared scale. L/G/Q denote Llama/Gemma/Qwen; numerals specify version and size.
Figure 12: Valid answer beginnings coexist with frequent budget exhaustion. Top: all 264 role/model/shot accuracies, each on 100 cases. Bottom: aggregate percentages over 1,100 cases per checkpoint; columns give demonstration counts. Invalid answers count as errors; description caps measure stopping behavior, not semantic accuracy.
Study
Analysis and target
Key finding
Behavioral robustness and semantic priors
Yang et al. (2022)
Input edits: semantics-preserving names
Plausible renaming induces failures; adversarial training improves robustness.
Gao et al. (2023)
Input counterfactuals: names
Names provide semantics and shortcuts; counterfactual inference improves robustness.
Wang et al. (2023)
Input edits: ReCode
Generation degrades under identifier, syntax, formatting, and docstring perturbations.
Ahmed et al. (2023) †
Input edits: masked code
Masked-operator predictions reflect code edits beyond frequency effects.
Hooda et al. (2024)
Input counterfactuals: predicates
Counterfactual code edits expose inconsistent handling of control/data flow and names.
Appendix
Table 12: Extended comparison of work on code-language-model behavior and representations. Groups follow Section 6 , with a final group for explanation methods and human comparison; studies are placed by their primary emphasis. The analysis column distinguishes input perturbations, representation readouts, and activation interventions. AST: abstract syntax tree; SAE: sparse autoencoder. † : cited as a preprint or technical note.
Figure 13: Strategies for selecting or creating negative examples, illustrated for the stepper role. Bold gold marks targets; headers give their roles. Selected negatives lack the target annotation, counterfactuals change the role, and renaming preserves it. Illustrative print readouts ( + ) immediately follow the statement or loop header containing the occurrence of interest, making that text available to causal attention. Δ marks edits, ellipses omit lines, and arrows mark wrapping. Examples come from annotated Python training data.
Figure 14: Strategies for selecting or creating negative examples, illustrated for the gatherer role. The six strategies vary the queried binding, context, update, or name. Renaming preserves the role and all non-target code; donor and recipient role sets are disjoint and their source groups differ. The miner uses target-masked code and logistic regression in three source-group-held-out folds within training. Added print readouts, highlighting and edit markers follow Figure 13 .
Figure 15: Different realizations of the most-wanted holder role. A guarded MWH updates only when an additional eligibility condition holds; a min/max MWH retains the extremal value observed so far; an MWH with a companion variable stores the selected object while another variable tracks the selection criterion.
Figure 16: Different realizations of the one-way flag role. The direct form uses a terminal assignment, typically inside a branch. The monotone form repeatedly combines the previous state with a new condition. Both allow state changes in only one direction.
Figure 17: Different realizations of the stepper role. A numeric iterator follows a regular progression controlling loop iteration. A predictive accumulator advances by a regular step to represent the position or state expected at the next iteration.
Figure 18: Different realizations of the walker role. An implicit iterator takes successive elements exposed by the loop construct. A structural iterator follows explicit links in the data structure.
Figure 19: Different realizations of the follower role. A follower derives its value from a changing master variable. Assignment order determines whether it captures the master’s value before or after a transition.
Figure 20: Different realizations of the most-recent holder role. A member holder receives successive collection elements. A loop-updated holder records the latest value satisfying an update condition. The stored value is replaced at every element or only at selected events, respectively.
Figure 21: Correctness-constrained calibration usually selects no intervention. (a) Ten of eleven completed cells select zero; the gray cell is unavailable after a failed zero-dose check. (b) In the sole nonzero cell, probe and SAE directions induce most-wanted holder on three and four of five eligible Test tasks, respectively, but each produces only one induction that passes the supplied tests. Bars include random controls on the same tasks; fractions show the small conditional denominators.
Figure 24: Annotation changes substantially exceed changes accompanied by passing tests. (a) Role disappearance for removal and induction for addition, with the joint passing outcome on the same baseline-defined cohort. (b) Joint outcome minus the matched random-control mean: random residual directions for probes and random SAE directions for SAE targets. (c) Intervention-minus-baseline correctness on that cohort. Rates weight target–task enrollments within each protocol; 95% intervals resample unique tasks 4,000 times, retaining all associated features, fits, and controls together. Normalized addition averages include selected zero doses; the stepper pilot uses Gemma 3 12B. The three protocol groups use different cohorts and perturbation scales.
Figure 25: Large structural shifts can describe a small surviving subset. Values reproduce Table 3 and extend it to Gemma 12B: syntax-frequency changes in percentage points among pairs with two clean Tree-sitter parses. Fractions show retained clean pairs over all target–task pairs. Markers follow Table 3 ; addition combines Test sufficiency and reference-negative cohorts. Gray cells have no eligible removal cohort.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act as evaluators in four tasks: picking their own solution from a pair, judging whether a single solution is their own, identifying which of two solutions a named model wrote, and judging quality blind. In the single-solution task, balanced accuracy is 49-58% for all 15 model-benchmark combinations, while raw accuracy (38-67%) mostly reflects how readily a model claims authorship. In the pairwise task, accuracy across 14 evaluator-opponent combinations correlates at r=0.93 with how often the evaluator's solution is longer. Attribution to a named model succeeds on some pairs and is consistently inverted on others. A rule-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and leaves ten of twelve re-tested results at chance; the other two follow a length difference it leaves, although a trained classifier still separates most normalized pairs. Claude Haiku's self-preference also disappears. We recommend reporting balanced accuracy, heuristic baselines, and label consistency.
Standard accuracy metrics cannot explain why LLMs handle variable tracking but fail on semantically equivalent loops. We study an internal lifecycle of code reasoning in which models first brew the answer, making it linearly recoverable many layers before it becomes self-decodable, and then diverge into one of four resolution outcomes: Resolved, Overprocessed, Misresolved, or Unresolved. Understanding this lifecycle matters because similar task accuracies can mask fundamentally different failure modes that surface-level evaluation cannot detect. We introduce a dual diagnostic framework pairing layer-wise linear probing with Context-Stripped Decoding (CSD) and apply it to six code-reasoning task families across 16 models spanning Qwen, Llama, and DeepSeek architectures. All four outcomes carry substantial mass in every task family: overall Resolved is only 41.5%, with multiple tasks below 30%. Controlled sweeps over structure, depth, and operators expose task-specific failure bottlenecks: Function Call Resolved plunges from 61.1% to 2.5% as call depth increases from one to three. Across architectures and scales, the brewing scaffold remains stable, with normalized brewing duration 24-42% across all 16 models, while resolution success varies with capability. This indicates that the scaffold is a stable empirical regularity across the tested decoder-only Transformer families, whereas resolution success covaries with capability, scale, and training. Code: https://github.com/euyis1019/llm-brewing
Siyue Chen, Yifu Guo, Yuquan Lu +9
1South China University of Technology · 2Sun Yat-sen University · 4Shanghai Jiao Tong University +5
The evaluation of large language models (LLMs) on coding tasks has primarily focused on performance metrics such as pass@k. As LLMs continue to advance, many models now meet baseline performance requirements, reducing the discriminative power of performance-based evaluation alone. Yet a key question remains largely unexplored: how do LLMs differ in their coding behavior? We propose CLIC (Code Learning for Identification and Comparison), a visual analytics approach that characterizes LLM coding behavior through token-frequency analysis. CLIC represents each code sample as a feature vector of token frequencies and trains an interpretable decision tree to separate two LLMs' code sets. Beyond classification accuracy, we define two new metrics: robustness, which measures whether the two LLMs remain distinguishable as their most-discriminative tokens are progressively removed, and concentration, which measures whether the difference is driven by a few dominant tokens or spread across many. Interpreting numerous pairwise comparisons (across LLM pairs, tasks, and tokenization levels) and tracing the full analytical chain form an inherently multi-scale, hypothesis-driven exploration task. We therefore develop an interactive visual analytics system to navigate the comparison landscape, identify pairs of interest, and drill down into discriminative tokens and their code contexts. Case studies comparing 10 LLMs across 22 Kaggle ML tasks reveal actionable insights for LLM selection and prompt engineering.