RSD: Moving Local Triangular Charts for Auditing Language-Model Hidden States

Existing bias auditing methods typically rely on model outputs, requiring costly benchmarks or judge models and potentially missing internal shifts that never appear in generated text. We propose a reference-based method that audits bias in hidden-state representations across related model variants, for example before and after fine-tuning. Because fine-tuning reshapes representation geometry, absolute hidden states are not directly comparable, so we encode each sentence by its similarities to a fixed set of anchor sentences, yielding relative representations in a shared comparison space. There we measure how target groups shift in their association with positive and negative attributes, a quantity we call the Representational Bias Shift $ΔB$. Across three model families and the WildGuardMix, DecodingTrust and ToxiGen benchmarks, $ΔB$ correlates with output-level bias change in 15 of the 18 settings we test, reaching $|r| = 0.84$ ($p < 0.001$) under full fine-tuning and becoming more model-dependent under parameter-efficient adaptation. Thresholding $ΔB$ detects checkpoints whose bias increased with ROC AUC between $0.65$ and $0.99$, and on WildGuardMix and DecodingTrust it separates them better than a SEAT-based baseline for all three families. $ΔB$ is also stable under changes to the anchor set, attribute sets and target templates. Our method requires no task-specific evaluation data and audits a model in about three minutes, using $3$-$50\times$ less compute than the output-level benchmarks considered here. We view it as complementary to output-based auditing rather than a replacement for it.

Knowing how often a language model fails does not explain where its errors concentrate. When auditors examine many explanations, the strongest observed pattern may arise by chance. We introduce Janus, a procedure for checking proposed error patterns before reporting them. Janus starts with a fixed list of yes/no properties of the examples being evaluated, such as whether the input is long. For each property, it compares the model's error rates on examples with that property and those without it. To see how large a difference can arise by chance, it repeats this calculation after shuffling the yes/no labels across examples without changing the group sizes. These shuffled properties are called decoys. A pattern is reported only if the size of its error difference meets a threshold set using decoys. On separate held-out examples, the same group must still have the higher error rate and the difference must meet a minimum, which was chosen in advance. In a controlled experiment, where the model must find a code in documents containing tables of staff, projects, and renewal codes, Janus confirms five related patterns of higher error rates on tasks requiring more lookups across tables. It also confirms a sixth pattern: lower error rates on examples with the needed information at the ends of the tables. In our samples from the MuSiQue and LongBench v2 public benchmarks, SliceLine finds groups with high error rates, while Janus reports no confirmed error patterns for the example properties we chose to test. For comparison, we use standard tests that shuffle errors and account for testing many candidates. With the same holdout check, they confirm two to six controlled patterns, depending on the test and threshold, and none on either benchmark. In simulations with no real error patterns, Janus reports false patterns more often than Benjamini-Hochberg, depending on the decoy count.

Removing the left context from a causal language model reveals a useful kind of boundary: an edge where the model processes the same right-hand tokens with little change. We turn this observation into prefix-removal probing and introduce Right Reset (RR), which measures preservation of the right-hand hidden-state trajectory. A dynamic program converts RR edge scores into variable-length chunks. On flattened text formed by concatenating topically similar records after deleting their separators and layout, RR recovers 47.7% of the original records as clean units, versus 25.9% for a BGE embedding boundary, the strongest tested conventional baseline without task-specific model training. The gain persists after rendering and OCR. Passive scores from the same Qwen3-4B layer and direct prompting of a same-scale instruction model perform substantially worse on flattened records. Across six language models, RR-selected cuts also undergo consistently less local output disruption than unselected candidate edges. An observed-token likelihood-ratio readout is competitive in some architectures, indicating that the central contribution is the intervention: context dependence itself can provide a boundary signal when surface structure is weak.