We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative ∂pz(p)=Az(p): its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term CKV=∑j(pj′−pj)εj, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.
Figures & tables
Figure 1: Construct, score, and check an attention intervention. The positional generator specifies a finite key displacement; exact softmax normalisation gives its local response, including any simultaneous value change. One baseline gradient supplies downstream scores for many candidates. Executed continuations measure the selected edits’ answer changes; a task check evaluates a proposed repair.
Task
Finite
Jacobian
Zero
Reduction vs. zero
Corrupt
0.02808
0.16028
0.05724
50.9% [48.2, 53.7]
Ordered
0.05786
0.21897
0.09073
36.2% [35.0, 37.5]
SST-2
0.03585
0.15426
0.06681
46.3% [43.1, 49.9]
Table 1: Prediction of executed positional effects. Downstream margin MAE on 256 held-out prompt sets per task. Intervals are paired 95% prompt-bootstrap intervals for the reduction relative to zero. All predictors use the same baseline gradient.
Model
Task
Layer
Separate
Quadratic
Exact
0.5B
Retrieval
18
0.001042
2.98×10−4
4.37×10−6
0.5B
SST-2
18
0.003099
5.99×10−4
2.64×10−5
1.5B
Retrieval
14
0.003342
0.001525
8.22×10−5
1.5B
Retrieval
21
0.107864
0.087683
0.007844
1.5B
SST-2
14
0.003194
0.001418
3.42×10−4
1.5B
SST-2
21
0.109616
0.080700
0.004510
Table 2: Joint-edit prediction across all six settings. Downstream margin MAE over 32 prompt pairs and eight joint edits each. Separate uses the exact finite-key response plus baseline-weighted values. Quadratic adds the leading mixed term; exact adds CKV . Complete controls and paired intervals are in Tables K.1 and K.2 .
Figure 2: Retaining the joint interaction improves executed-effect prediction. Left: downstream MAE at the three measured edit strengths on 1.5B retrieval, layer 21; lines connect measured means on a log scale. Right: the six full-replacement settings, with each MAE divided by the separate predictor’s MAE in that setting. Absolute values are in Table 2 . Predictors share a baseline gradient. Points show reported means; paired 95% intervals are reported in Table K.2 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: The derivative determines the tangent; its integral gives the finite response. Left: a single rotary plane, with the exact chord and the tangent step at the same displacement. Right: an analytic two-key example with scores (sinθ,0) and scalar values (1,0) . The exact readout change is sigmoid(sinθ)−1/2 ; the positional Jacobian gives θ/4 . This illustration separates the mathematical issue from downstream model behaviour; it is not an empirical model measurement.
Figure F.1: Downstream prediction across six fixed settings at full replacement. Bars plot the reported margin MAEs over 32 prompt pairs with eight joint span edits each (Table K.1 ). Axes are scaled separately, and exact-joint MAEs are labelled. All predictors share the same baseline downstream gradient; targets are independently executed margin changes. The quadratic comparator retains the exact key response and leading mixed term. Paired 95% intervals are reported in Table K.2 .
Figure F.2: Intervention-strength diagnostic on 1.5B retrieval layer 21, using the same 32 prompt pairs and eight spans at each strength. A: Exact-joint and quadratic mean downstream errors at the three evaluated strengths; lines connect measured points on a log scale. B: The same predictors at α=0.1 on a linear scale. The plot uses reported summary means. The paired 95% interval for their small-strength difference is [−1.09×10−4,−2.88×10−5] .
Figure F.3: Native local-response latency for the same eight candidates on an A100. Bars show the reported medians over five timing rounds. Scoring-only latency assumes prepared statistics. The other groups directly time preparation plus evaluation at one or three strengths; three strengths share one preparation in both methods. Model execution and gradient acquisition are excluded throughout. Both dense and sparse methods evaluate the exact local response.
Branch
Generated expression
Checks passed
Ordinary
lookup(value)
0/6
Source emphasis
current(value)
1/6
Installed no-op
lookup(value)
0/6
Ranked attention edit
legacy(current(value))
6/6
Appendix
Table H.1: Recorded outcomes on the same six fixed checks. Ordinary generation and the attention branch use the same prompt. Source emphasis supplies the full current API source. The installed no-op reproduces ordinary output token for token. Expressions omit the mechanically added API namespace.
Figure J.1: Summary of the initial single-layer experiment. A: Reported mean downstream prediction errors (Table J.1 ). B: Mean errors for predicting the executed four-cell interaction with G(C2) and G(CKV) ; the dashed line is the mean absolute executed interaction. Bars are aggregate summaries over the 32 prompts and eight spans.
We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix ΔMi satisfying yi=μi+ui⊤ΔMi, where μi is the uniform mean of the attended values and ui is the augmented query input. The construction applies the classical exponential divided difference ρ=φ1 to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same function connects the RoPE generator to exact positional finite differences. We derive a tokenwise formula for the error of reusing one query's matrix and prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout. Reconstruction checks and frozen-reuse calibration on one pretrained Qwen2.5-0.5B layer verify the representation and quantify the correction required when one query's matrix is reused.
Julie Huang, Maggie Chlon, Leon Chlon
Hassana Labs · Department of Information Engineering, Oxford University
Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.
KV-cache reuse can reduce inference cost in retrieval-augmented generation and agentic systems, but cached contexts may become stale when retrieved knowledge, working memory, or user state is edited. Under causal self-attention, even a local edit can affect downstream KV states. A full re-prefill reliably restores consistency but is costly, whereas refreshing only the edited span can leave downstream dependencies stale. We formulate in-place repair as budgeted recomputation and compare training-free position-selection policies on a factual RAG benchmark with matched direct and derived edits. Across three model families, all policies repair direct cases, but derived cases clearly separate them. At the primary budget, a contiguous edit-local window recovers at least 0.94 of the post-edit answer margin and substantially outperforms attention-based, KV-deviation, and structural selectors. Mechanistic analysis shows that position sets effective under clean-state transplantation can fail under actual recomputation because scattered positions inherit surrounding staleness. The edit-local advantage also depends on adjacency and largely disappears when the answer-bearing text moves downstream. Because answer-relevant edits almost always corrupt model behavior, failure severity is difficult to predict, and repair is 13-21 times faster than full re-prefill, our results support unconditional edit-local repair when the dependent text remains adjacent to the edit.
Mingyang Mao, Wyatt Mackey, Xiaomin Lin
University of South Florida, Electrical and Computer Engineering · DEVCOM Army Research Laboratory