Kinematic Hypothesis Context: Embodiment-Conditioned Learned Video Coding for Bandwidth-Limited Robot Policy Uplinks
Organizations: Tsinghua University, Beijing, China
Abstract
Robots increasingly run large vision-language-action policies on remote servers, making the camera uplink the bottleneck. Existing video codecs spend bits on content the receiver could predict: the robot's own motion. We introduce the kinematic hypothesis context, a zero-rate conditioning signal that both ends build from joint states, the robot model and the camera calibration. Ego-motion homographies, per-link robot reprojections, an end-effector plane and a background memory warp the previous reconstruction into hypotheses that condition a learned video codec. We prove that history-only and global-pose conditioning with a limited receptive field cannot access this information under large motion, and that coding gains become task gains at the policy's success cliff. On LIBERO, the context saves 23.9% bit rate over a capacity-matched history-conditioned codec and 40-56% over x264, x265 and SVT-AV1, and needs 23.2% fewer bits than DCVC-RT at equal policy action deviation; trained to lower rates, it matches DCVC-RT in PSNR. In closed loop at equal rate, OpenVLA-OFT success nearly doubles from 22.8% to 43.1% at 6 kbps (72.6% with lower-rate training, 23.6 points above DCVC-RT), pi_0.5 success rises from 55.5% to 77.4%, and the gain grows under packet loss. The context transfers across three codec hosts and to real robot data.
Figures & tables
| Type | Anchor | PSNR BD-rate (%) | MS-SSIM (%) | LPIPS (%) | Action dev. (%) | ||
|---|---|---|---|---|---|---|---|
| Both cameras | Wrist | Third-person | Both | Both | Both | ||
| Capacity- | HIST (history context) | 23.9 [ 25.0, 22.8] | 12.3 [ 12.9, 11.6] | 37.9 [ 39.8, 36.0] | 18.2 [ 19.5, 16.8] | 24.6 [ 25.5, 23.7] | 26.2 [ 27.6, 24.9] |
| matched | HIST2 (two references) | 26.0 [ 27.2, 25.0] | 15.1 [ 15.8, 14.5] | 39.6 [ 41.4, 37.7] | 17.6 [ 19.2, 15.9] | – | – |
| SHUF (shuffled kinematics) | 24.8 [ 25.9, 23.6] | 12.1 [ 12.8, 11.5] | 40.0 [ 41.6, 38.4] | 19.2 [ 20.4, 17.9] | – | – | |
| GVEC (global pose vector) | 23.3 [ 24.7, 21.8] | 10.7 [ 11.5, 9.8] | 38.5 [ 40.5, 36.4] | 19.4 [ 20.8, 17.9] | – | – | |
| Neural | DCVC-RT [ 17 ] | +7.5 [ 0.5, 16.4] | +7.2 [3.1, 11.7] | 0.2 [ 11.5, 13.1] | +9.3 [2.0, 17.4] | 13.6 [ 20.8, 5.1] | 23.2 [ 26.5, 19.8] |
| Variant | vs. full KHC (%) | vs. HIST (%) | ms |
|---|---|---|---|
| Full KHC ( ) | – | 9.99 [ 10.8, 9.2] | 41.3 |
| w/o plane-sweep slots | +8.96 [8.2, 9.7] | 3.03 [ 4.0, 2.0] | – |
| w/o robot-layer slot | +26.39 [24.5, 28.3] | +11.76 [10.1, 13.3] | – |
| w/o end-effector plane | +5.32 [4.5, 6.2] | 6.40 [ 7.4, 5.2] | – |
| w/o background memory | +5.83 [5.2, 6.5] | 5.73 [ 6.7, 4.7] | – |
| 3.45 [ 4.0, 2.8] | 12.62 [ 13.6, 11.6] | 41.3 |
| Budget | Uplink | Success (%) | Rate (kbps) | SKIP |
|---|---|---|---|---|
| – | Uncompressed RAW | 96.94 [96.0, 97.7] | 7496 | – |
| 6 kbps | HIST-skip | 22.75 [20.8, 24.9] | 5.98 | 0.47 |
| KHC-skip, identity-only | 30.38 [28.2, 32.7] | 5.97 | 0.42 | |
| KHC-skip (ours) | 43.12 [40.7, 45.6] | 6.07 | 0.46 | |
| x265 (CBR) | 32.44 [30.2, 34.8] | 9.85 | – | |
| 8.3 kbps | HIST-skip | 57.06 [54.6, 59.5] | 8.15 | 0.24 |
| Uplink | Context | SKIP pose | Budget | Enc. (ms) | Param. (M) | Succ. (%) |
|---|---|---|---|---|---|---|
| KHC-skip (ours) | ✓ | ✓ | ✓ | 26.7 | 7.0 | 72.4 |
| KHC-LR-skip (ours) | ✓ | ✓ | ✓ | 26.7 | 7.0 | 86.0 |
| HIST-skip | – | – | ✓ | 23.4 | 7.0 | 55.8 |
| DCVC-RT | – | – | ✓ | 12.8 | 20.7+45.7 | 66.6 |
| HIST, | – | – | ✓ | – | 7.0 | 62.6 |
| x265, CBR | – | – | 49% | – | – | 64.8 |
| Anchor | PSNR | LPIPS | Action dev. | |||
|---|---|---|---|---|---|---|
| Full | Low | Full | Low | Full | Low | |
| HIST-LR (same recipe) | 27.6 | 28.7 | 28.7 | 31.1 | 29.7 | 31.1 |
| KHC (E2 model) | 5.5 | 5.1 | 1.5 | 1.9 | 2.5 | 2.4 |
| DCVC-RT [ 17 ] | +0.1 ∘ | 5.7 ∘ | 24.7 | 33.2 | 30.1 | 34.2 |
| VTM-23 (4:2:0) | +16.1 | +2.0 ∘ | 4.9 ∘ | 18.2 | – | – |
| VTM-23 (4:4:4) | +22.7 | +6.0 | +2.9 ∘ | 12.2 | 2.0 ∘ | 8.7 |
| Uplink | 6 kbps | 8.3 kbps | 10.7 kbps |
|---|---|---|---|
| KHC-LR-skip (ours) | 72.6 | 86.0 | 88.9 |
| HIST-LR-skip (same recipe) | 49.0 (+23.6 ∗ ) | 80.3 (+5.8 ∗ ) | 85.6 (+3.3 ∗ ) |
| KHC-skip (E2 model) | 43.6 (+29.0 ∗ ) | 72.4 (+13.6 ∗ ) | 88.9 (+0.0) |
| DCVC-RT, same controller | 49.0 (+23.6 ∗ ) | 66.6 (+19.4 ∗ ) | 69.4 (+19.5 ∗ ) |
| HIST-skip, | 53.1 (+19.5 ∗ ) | 62.6 (+23.4 ∗ ) | 71.0 (+17.9 ∗ ) |
| Measured rate of KHC-LR (kbps) | 5.81 | 7.69 | 9.63 |
Video Compression
Video compression reduces the bits needed to store or transmit moving images while balancing visual quality, computational cost, and downstream usefulness.
See all 80 papers