stat.MLSep 12, 2026

Exact Finite Attention Responses From RoPE Derivatives

Authors: Julie Huang, Maggie Chlon, Gregory Gutin, Leon Chlon

Organizations: Hassana Labs · Department of Computer Science, Royal Holloway, University of London · University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative ∂pz(p)=Az(p)\partial_p z(p) = A z(p): its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term CKV=∑j(pj′−pj) εjC_{KV} = \sum_j (p'_j - p_j)\,\varepsilon_j, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.

Figures & tables

Figure 1: Construct, score, and check an attention intervention. The positional generator specifies a finite key displacement; exact softmax normalisation gives its local response, including any simultaneous value change. One baseline gradient supplies downstream scores for many candidates. Executed continuations measure the selected edits’ answer changes; a task check evaluates a proposed repair.
TaskFiniteJacobianZeroReduction vs. zero
Corrupt0.028080.160280.0572450.9% [48.2, 53.7]
Ordered0.057860.218970.0907336.2% [35.0, 37.5]
SST-20.035850.154260.0668146.3% [43.1, 49.9]
Table 1: Prediction of executed positional effects. Downstream margin MAE on 256 held-out prompt sets per task. Intervals are paired 95% prompt-bootstrap intervals for the reduction relative to zero. All predictors use the same baseline gradient.
ModelTaskLayerSeparateQuadraticExact
0.5BRetrieval180.0010422.98×10−42.98{\times}10^{-4}4.37×10−64.37{\times}10^{-6}
0.5BSST-2180.0030995.99×10−45.99{\times}10^{-4}2.64×10−52.64{\times}10^{-5}
1.5BRetrieval140.0033420.0015258.22×10−58.22{\times}10^{-5}
1.5BRetrieval210.1078640.0876830.007844
1.5BSST-2140.0031940.0014183.42×10−43.42{\times}10^{-4}
1.5BSST-2210.1096160.0807000.004510
Table 2: Joint-edit prediction across all six settings. Downstream margin MAE over 32 prompt pairs and eight joint edits each. Separate uses the exact finite-key response plus baseline-weighted values. Quadratic adds the leading mixed term; exact adds CKVC_{KV} . Complete controls and paired intervals are in Tables K.1 and K.2 .
Figure 2: Retaining the joint interaction improves executed-effect prediction. Left: downstream MAE at the three measured edit strengths on 1.5B retrieval, layer 21; lines connect measured means on a log scale. Right: the six full-replacement settings, with each MAE divided by the separate predictor’s MAE in that setting. Absolute values are in Table 2 . Predictors share a baseline gradient. Points show reported means; paired 95% intervals are reported in Table K.2 .
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: The derivative determines the tangent; its integral gives the finite response. Left: a single rotary plane, with the exact chord and the tangent step at the same displacement. Right: an analytic two-key example with scores (sin⁡θ,0)(\sin\theta,0) and scalar values (1,0)(1,0) . The exact readout change is sigmoid⁡(sin⁡θ)−1/2\operatorname{sigmoid}(\sin\theta)-1/2 ; the positional Jacobian gives θ/4\theta/4 . This illustration separates the mathematical issue from downstream model behaviour; it is not an empirical model measurement.
Figure F.1: Downstream prediction across six fixed settings at full replacement. Bars plot the reported margin MAEs over 32 prompt pairs with eight joint span edits each (Table K.1 ). Axes are scaled separately, and exact-joint MAEs are labelled. All predictors share the same baseline downstream gradient; targets are independently executed margin changes. The quadratic comparator retains the exact key response and leading mixed term. Paired 95% intervals are reported in Table K.2 .
Figure F.2: Intervention-strength diagnostic on 1.5B retrieval layer 21, using the same 32 prompt pairs and eight spans at each strength. A: Exact-joint and quadratic mean downstream errors at the three evaluated strengths; lines connect measured points on a log scale. B: The same predictors at α=0.1\alpha=0.1 on a linear scale. The plot uses reported summary means. The paired 95% interval for their small-strength difference is [−1.09×10−4,−2.88×10−5][-1.09\times 10^{-4},-2.88\times 10^{-5}] .
Figure F.3: Native local-response latency for the same eight candidates on an A100. Bars show the reported medians over five timing rounds. Scoring-only latency assumes prepared statistics. The other groups directly time preparation plus evaluation at one or three strengths; three strengths share one preparation in both methods. Model execution and gradient acquisition are excluded throughout. Both dense and sparse methods evaluate the exact local response.
BranchGenerated expressionChecks passed
Ordinarylookup(value)0/6
Source emphasiscurrent(value)1/6
Installed no-oplookup(value)0/6
Ranked attention editlegacy(current(value))6/6
Appendix
Table H.1: Recorded outcomes on the same six fixed checks. Ordinary generation and the attention branch use the same prompt. Source emphasis supplies the full current API source. The installed no-op reproduces ordinary output token for token. Expressions omit the mechanically added API namespace.
Figure J.1: Summary of the initial single-layer experiment. A: Reported mean downstream prediction errors (Table J.1 ). B: Mean errors for predicting the executed four-cell interaction with G(C2)\mathcal{G}(C_{2}) and G(CKV)\mathcal{G}(C_{KV}) ; the dashed line is the mean absolute executed interaction. Bars are aggregate summaries over the 32 prompts and eight spans.

Explore similar work

CardsList

  1. RoPE attention is an exact forward-pass gradient step with softmax intact

    Sep 6, 2026Julie Huang, Maggie Chlon, Leon ChlonSoftmax AttentionRotary Positional Embeddings

    Hassana Labs · Department of Information Engineering, Oxford University

  2. WavePrune: One period is often enough for RoPE

    Oct 3, 2026Guancheng Du, Luotian Huang, Shaowen Wang +2Rotary Positional EmbeddingsPositional Encoding

    Tsinghua University

  3. Contiguity, Not Importance: Budgeted Repair of Stale KV Caches After Document Edits

    Sep 16, 2026Mingyang Mao, Wyatt Mackey, Xiaomin LinRetrieval-Augmented GenerationKV Caching

    University of South Florida, Electrical and Computer Engineering · DEVCOM Army Research Laboratory