Encoding, Decodability, and Use: A Five-Layer Framework for Benchmark Measurement Audits with Evidence from UAV Intent Recognition
Organizations: Shandong Management University, Jinan 250357, China. · Tsinghua University, Beijing 100084, China.
Abstract
Benchmark accuracy is routinely read as evidence that a model understands the construct the benchmark measures. We propose and instantiate on AeroIntent a five-layer framework that decomposes that reading into observation, representation, decoding, use, and behaviour. The framework separates encoding, decodability, and use as distinct levels. In the instantiation, the observation layer is unreliable on 7 of the 20 Drone programs and floors the direct visual correlate of radial velocity at a 0.26% scale change. The representation layer exposes two task-designed RGB features with nominal signals. RAFT flow radial bias reaches (), and bounding-box optical-flow divergence reaches (). No multi-feature readout gives a fold-mean test correlation above the cluster minimum detectable effect (MDE), and the largest positive global out-of-sample correlation is . The decoding layer sets the MDE at at the multi-feature Bonferroni threshold (, power 0.80), and the observed has power 0.00. Two injection experiments establish the MDE's condition from both sides: aligned injections are detected at or above the simulated power, and orthogonal injections are not detected. The use layer finds accuracy consistent with program memory (program-identity 0.937/0.597/0.210) and multi-region static cues. The behaviour layer is consistent with accuracy not requiring a radial-velocity readout (appearance 0.6552 vs bounding-box trajectory 0.4080, cluster-level paired , ).
Figures & tables
| Layer | Failure mode | Audit method | Evidentiary type |
|---|---|---|---|
| 1a Observation: reliability | Reconstruction of disagrees with independent state | Reconstruction vs. analytic state; per-sequence sign agreement | Observational (Obs.) |
| 1b-oracle Capacity: window contains signal | Window does not contain the signal through the oracle channel | Oracle descriptor from same window | Obs. |
| 1b-visual-floor Capacity: visual correlate | Direct visual correlate of is below the window’s noise floor | Physical floor analysis of the visual correlate | Obs. |
| 2 Representation | Signal not recoverable by the linear readout family | Readout family: linear (single-feature, PCA+Ridge), kernel, nonlinear | Obs. |
| 3 Decoding | Readout family too weak; sample size too small; cluster structure unhandled | MDE under the actual scheme; cluster permutation; per-fold stability | Statistical (Stat.) |
| 4 Use | Classifier uses program memory, static cues, or other shortcuts | Program classifier; program-label shuffle; per-program accuracy; region ablation | Associational (Assoc.) |
| Target | OOS | cluster | null 90% |
|---|---|---|---|
| Reconstruction (V1) | 0.353 | ||
| Analytic (V3) | 0.874 | ||
| Paired (V1 V3) | 0.636 |
| Layer | Evidence | Type | Strength |
|---|---|---|---|
| 1a | 13/20 sign agreement; two outliers | Obs. | Strong |
| 1b-oracle | Oracle control | Obs. | Strong |
| 1b-visual-floor | Scale-factor floor 0.26% | Obs. | Moderate |
| 2 | Nominal single-feature signals; no multi-feature readout | Obs. | Moderate |
| 3 | MDE under cluster permutation | Stat. | Strong |
| 4 | Program classifier; shuffle; region ablation | Assoc. | Moderate |
| Sequence | V1 | V3 |
|---|---|---|
| boundary_probe ∗ | ||
| fan_scanning_recon ∗ | ||
| cruise_transit_v5 | ||
| evasive_escape | ||
| feint_attack_270 | ||
| feint_attack_90 |
| Family | Feature | perm | programs | |
| B (RAFT) | flow radial bias | 0.030 | 19/22 | |
| A (Farneback) | flow radial bias | 0.234 | 14/22 | |
| A (Farneback) | mean flow magnitude | 0.180 | 8/22 | |
| A (frame diff) | ring-3 frame difference | 0.176 | 10/22 | |
| C (task-designed) | bounding-box divergence | 0.029 | 11/21 | |
| C (task-designed) | bounding-box scale change | 0.813 | — |
| Representation | PCA | train | test | folds | global OOS | perm |
|---|---|---|---|---|---|---|
| RAFT flow | none | 5/22 | 0.836 | |||
| Farneback flow | none | 9/22 | 0.828 | |||
| Frame diff | none | 3/22 | 0.934 | |||
| ResNet-18 | none | 9/22 | 0.018 | |||
| ResNet-18 | 2 | 9/22 | 0.747 | |||
| ResNet-18 | 10 | 13/22 | 0.55 |
| Descriptor | Dims | with full |
|---|---|---|
| Full-trajectory GPS shape | 12 | |
| 16-frame GPS shape (oracle) | 10 | |
| 16-sample mean of | 16 | |
| RAFT flow radial bias (single) | 1 | |
| bounding-box divergence (single) | 1 | |
| bounding-box scale change (single) | 1 |
| Significance level | Family | MDE |
|---|---|---|
| 0.0025 | A (Farneback + frame diff, 20 features) | |
| 0.0045 | B (RAFT, 11 features) | |
| 0.00714 | multi-feature Bonferroni (7 primary configs) | |
| 0.025 | C (task-designed, 2 features) | |
| 0.05 | uncorrected |
| simulated | observed | simulated | observed | ||
|---|---|---|---|---|---|
| 0.20 | 0.170 | 0.000 | 0.03 | 0.000 | 0.00 |
| 0.30 | 0.239 | 0.000 | 0.29 | 0.000 | 0.04 |
| 0.45 | 0.335 | 0.915 | 0.95 | 0.000 | 0.58 |
| 0.60 | 0.429 | 1.000 | 1.00 | 0.810 | 0.99 |
| 0.70 | 0.495 | 1.000 | 1.00 | 1.000 | 1.00 |
| Representation | Sequence-disjoint | Config-disjoint | Program-disjoint |
|---|---|---|---|
| Bounding-box trajectory features | 0.943 | 0.943 | 0.408 |
| ResNet-18 mean-pooled | 0.847 | 0.830 | 0.648 |
| VideoMAE-base CLS | 0.483 | 0.489 | 0.392 |
| GRU + attention | 0.761 | — | 0.415 |
| Random (5 classes) | 0.200 | 0.200 | 0.200 |
| Majority class | 0.273 | 0.273 | 0.273 |
| Representation | program acc | chance (program) | intent acc | shuffle null |
|---|---|---|---|---|
| (config-disjoint) | (program-disjoint) | (program-disjoint) | ||
| Bounding-box trajectory | 0.937 | 0.045 | 0.408 | 0.196 ( ) |
| ResNet-18 | 0.597 | 0.045 | 0.648 | 0.193 ( ) |
| VideoMAE | 0.210 | 0.045 | 0.392 | 0.189 ( ) |
| Intent | Original | Bg-only | Target-only | Supported by |
|---|---|---|---|---|
| transit | 0.783 | 0.870 | 0.804 | background |
| reconnaissance | 0.688 | 0.708 | 0.688 | both |
| probe | 0.667 | 0.667 | 0.458 | background |
| escape | 0.500 | 0.500 | 0.625 | target |
| strike | 0.525 | 0.325 | 0.375 | both |
| Accuracy | 0.6552 | 0.638 | 0.609 |
| Layer | Evidence | Verdict | Type |
|---|---|---|---|
| 1a | sign agreement 13/20; disagreements 7/20, including boundary_probe ( vs ) and fan_scanning_recon ( vs ). | unreliable (7/20) | Obs. |
| 1b-oracle | oracle window control . | signal present | Obs. |
| 1b-visual-floor | scale-factor floor 0.26%; , . | visual floored | Obs. |
| 2 | RAFT flow radial bias ( ; family B); bounding-box divergence ( ; family C); multi-feature OOS . | mixed (aligned only) | Obs. |
| 3 | MDE ( , aligned); (families A/B/C); ( ); observed , power 0.00; aligned injection meets or exceeds simulated power, orthogonal injection at not detected. | excludes aligned MDE; not aligned MDE; not orthogonal | Stat. |
| 4 | program-identity 0.937/0.597/0.210; shuffle ; bg-only 0.638, target-only 0.609. | program memory; static cues | Assoc. |
Benchmark Auditing
Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.
See all 109 papers