Auditing Targetless Camera-LiDAR Extrinsic Correction before Recursive Use
Organizations: Auckland University of Technology, Auckland 1024, New Zealand
Abstract
Camera-LiDAR fusion repeatedly uses an extrinsic transform in projection, association, and recursive state estimation. We study whether a targetless correction should be admitted to a downstream consumer only after separate audits of correction quality, confidence ranking, and recursive risk. Our log-disjoint protocol evaluates these stages under synthetic extrinsic drift on Argoverse 2 and treats cross-dataset and odometry experiments as bounded diagnostics. On 27 test-split logs, single-frame Canny alignment is harmful in the mild tier. Multi-frame alignment improves severe-tier rotation by 2.52 degrees but worsens translation by 0.177 m; its declared joint error-magnitude improvement is also negative (-0.158, 95% CI [-0.223, -0.098]), so the rotation gain is not a six-degree-of-freedom correction success. A zero-injection check moves away from the true extrinsic, and solver-disagreement AUROC on selected splits does not yield a validated nonzero-coverage risk-coverage operating point. In a 15-log final system role, the locked conservative policy admits no candidate and equals Raw, whereas Always-correct increases trajectory error. These results support an auditable separation of correction, confidence, and recursive contamination; they do not establish a deployable calibration gate, positive odometry or SLAM improvement, map consistency, or closed-loop safety.
Figures & tables
| Data role / fixed sample | Boundary and prohibited inference |
|---|---|
| Stage 1 formal: AV2 test, 27 logs; every 3rd frame; 1,435 frames/tier/condition | True : weak labels and offline metrics only; no test-derived threshold or decision |
| Learned split: same 27 logs, fixed 18/9 train/holdout; 473 holdout frames/tier | Heatmap targets use on train only; holdout is not used for tuning |
| Exploratory ranking: AV2 train/val, 60 logs; post-hoc 30/30 split | Severity is for stratification only; ranking/risk scans cannot select the system threshold |
| System audit: 3/15/15 development/calibration/final logs; 100 sweeps, frames, target 50 | Calibration locks; final is evaluation only; logged ego pose is offline/oracle compensation |
| External diagnostic: 170/850 nuScenes Trainval scenes; no locked manifest hash | Plumbing/failure-pattern diagnostic only; no cross-domain generalization claim |
| Tier | Points | Raw Corr. (rot.) | Rot. impr. [95% CI] | Trans. impr. (m) [95% CI] | Joint impr. [ m] | AUROC [95% CI] |
|---|---|---|---|---|---|---|
| Mild | Edge | [ ] † | [ ] † | [ ] † | [ ] | |
| Mild | Random | [ ] † | [ ] † | [ ] † | [ ] | |
| Severe | Edge | [ ] | [ ] † | [ ] † | [ ] ‡ | |
| Severe | Random | [ ] † | [ ] † | [ ] † | [ ] |
| Method | Tier | Rot. impr. [95% CI] | Trans. impr. (m) [95% CI] | Joint impr. [ m] | AUROC [95% CI] |
|---|---|---|---|---|---|
| Canny, single-frame a | Mild | [ ] | [ ] ‡ | [ ] | |
| Canny, single-frame a | Severe | [ ] | [ ] ‡ | [ ] | |
| Canny, multi-frame b | Mild | [ ] | [ ] ‡ | [ ] | |
| Canny, multi-frame b | Severe | [ ] | [ ] ‡ | [ ] | |
| Learned, single-frame a | Mild | [ ] | [ ] ‡ | [ ] | [ ] |
| Learned, single-frame a | Severe | [ ] | [ ] ‡ | [ ] | [ ] |
| Tier | Method | % Improved | Basin-consistency AUROC [95% CI] | Legacy (cost-drop) AUROC [95% CI] |
|---|---|---|---|---|
| Correct | Multi-frame | n/a (no frame improved) | n/a | |
| Correct | Single-frame | n/a (no frame improved) | n/a | |
| Mild | Multi-frame | [ ] ∗ | [ ] | |
| Mild | Single-frame | [ ] | [ ] | |
| Severe | Multi-frame | [ ] ∗ | [ ] | |
| Severe | Single-frame | [ ] | [ ] |
| Tier | Method | % Improved | Basin-consistency AUROC [95% CI] | Legacy (cost-drop) AUROC [95% CI] |
|---|---|---|---|---|
| Correct | Multi-frame | n/a (no frame improved) | n/a | |
| Correct | Single-frame | n/a (no frame improved) | n/a | |
| Mild | Multi-frame | [ ] ∗ | [ ] | |
| Mild | Single-frame | [ ] | [ ] | |
| Severe | Multi-frame | [ ] ∗ | [ ] ∗ | |
| Severe | Single-frame | [ ] | [ ] |
| Metric | Raw | Oracle | Always-correct | Guarded |
|---|---|---|---|---|
| ATE RMSE (m) | ||||
| Rotation RMSE (deg) | ||||
| 10-frame RPE rot. (deg) | ||||
| 10-frame RPE trans. (m) | ||||
| 10-frame joint RPE | ||||
| Reprojection RMSE (px) |
| Evaluation item | Calibration | Final | Decision |
|---|---|---|---|
| Logs / tier rollouts | 15 / 45 | 15 / 45 | log-disjoint |
| Candidate observations | 2,748 | 2,748 | audit only |
| Usable frames | min. 30 | ||
| Long-horizon logs ( ) | – | 14/15 | 1 short-log |
| Guarded admissions | locked 0 | 0 | negative control |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Role | Dataset / split | Unit, stride, perturbation | N | Split identity |
|---|---|---|---|---|
| Stage 1 formal (Tables II and III Canny; formal basin-consistency) | AV2 Sensor Dataset, official test split | per-frame, every 3rd LiDAR frame; 3 tiers (correct/mild/severe) sampled once per (log, frame, tier) via a deterministic hash seed | 27 logs, 1,435 frames/tier/condition | Pre-specified 27-log test-split list |
| Learned-corrector training role | Same 27 formal logs (fixed 18-log subset) | per-frame, same tier sampling as above | 18 of the 27 logs above | Fixed log-level training partition |
| Learned-corrector evaluation role | Same 27 formal logs (disjoint 9-log subset) | per-frame, same tier sampling as above | 9 of the 27 logs above; 473 frames/tier | Fixed log-level evaluation partition |
| AV2 60-log exploratory pool (basin-consistency, full- exploratory) | AV2 Sensor Dataset, official train + val splits (30 train, 15 val_calibration, 15 val_eval) | per-frame, same tier sampling as above | 60 logs | Pre-specified train/validation log membership and source-split labels |
| Selective-risk calibration / evaluation split | Post-hoc 30/30 partition of the same 60-log pool above (no new logs) | same as the 60-log pool | 30 + 30 | Post-hoc calibration/evaluation partition of the same pool |
| System audit: development, calibration, final | AV2 Sensor Dataset, official train and validation splits | Requested 100 sweeps; at least 30 usable consecutive frames and a 50-frame target; 3 tiers sampled once per log and evaluated with four paired arms | 3 + 15 + 15 = 33 logs | Fixed role manifest with log-disjoint development, calibration, and final roles |
| Source | Tier | Logs | Rotation [95% CI] | Translation (m) [95% CI] | Joint ( m) [95% CI] |
|---|---|---|---|---|---|
| Table II edge | Mild | 27 | [ ] | [ ] | [ ] |
| Table II random | Mild | 27 | [ ] | [ ] | [ ] |
| Table II edge | Severe | 27 | [ ] | [ ] | [ ] |
| Table II random | Severe | 27 | [ ] | [ ] | [ ] |
| Canny single | Mild | 9 | [ ] | [ ] | [ ] |
| Canny single | Severe | 9 | [ ] | [ ] | [ ] |
| Source | Tier | Variant | Rotation (deg) | Translation (m) | Joint ( m) | |||
|---|---|---|---|---|---|---|---|---|
| Mean [95% CI] | Mean [95% CI] | Mean [95% CI] | ||||||
| Formal Canny | Mild | edge | [ ] | .006 | [ ] | .006 | [ ] | .006 |
| Formal Canny | Mild | control | [ ] | .006 | [ ] | .006 | [ ] | .006 |
| Formal Canny | Severe | edge | [ ] | 1.000 | [ ] | .006 | [ ] | .006 |
| Formal Canny | Severe | control | [ ] | .006 | [ ] | .006 | [ ] | .006 |
| Held-out Canny | Mild | edge | [ ] | .031 | [ ] | .012 | [ ] | .018 |
| Source | Tier | Variant | Corrected RMSE (px) | True RMSE (px) | Corrected 3px | True 3px | Corr.-true median (px) |
|---|---|---|---|---|---|---|---|
| Formal Canny | Mild | edge | 12.13 | 15.32 | .5323 | .4639 | 147.13 |
| Formal Canny | Mild | control | 21.26 | 35.90 | .4473 | .3277 | 217.25 |
| Formal Canny | Severe | edge | 12.48 | 15.32 | .5257 | .4639 | 167.49 |
| Formal Canny | Severe | control | 21.69 | 35.90 | .4410 | .3277 | 237.72 |
| Canny multi | Mild | 14.32 | 15.21 | .4793 | .4644 | 92.57 | |
| Canny multi | Mild | single | 10.58 | 13.33 | .5457 | .4640 | 148.77 |
| Audit role | Unit | Observed data or metric | Interpretation |
|---|---|---|---|
| Vicon checkerboard | 30 keypoints | Projection finite; GT ID pairs TP/FP/FN ; adversarial case | Evaluator smoke only |
| Xiaoliang target example | 36 image/cloud frames | 48 image points; 100 LiDAR points/frame; 33 paired target-model frames | Target-frame format; no point-level cross-sensor IDs |
| A9 north | 29 sensor-frames | 691 camera detections; 410 LiDAR detections; 319 labels; all-pairs precision , recall | Detection-level label audit |
| A9 south | 29 sensor-frames | 699 camera detections; 525 LiDAR detections; 0 labels; all-pairs precision , recall n/a | No positive labels in this channel |
| A9 total | 58 sensor-frames | 1,390 camera detections; 935 LiDAR detections; 319 labels; all-pairs precision , recall | Not method P/R and not feature-track P/R |
| Experiment / role | Unit | Reprojection RMSE (px) | Association P/R/F1 | Coverage |
|---|---|---|---|---|
| Synthetic AprilGrid-like / raw | 1 frame, 46 reference pairs | 22.584 | 0/0/0 | 0 |
| Synthetic AprilGrid-like / corrected | 1 frame, 46 reference pairs | 3.068 | 0.9111/0.8913/0.9011 | 0.8913 |
| Synthetic AprilGrid-like / true | 1 frame, 46 reference pairs | 0.504 | 1/1/1 | 1 |
| Xiaoliang checkerboard / raw | 32 real target images, 1,536 pairs | 38.168 | 0/0/0 | 0 |
| Xiaoliang checkerboard / corrected | 32 real target images, 1,536 pairs | 3.897 | 1/1/1 | 1 |
| Xiaoliang checkerboard / true PnP reference | 32 real target images, 1,536 pairs | 0.609 | 1/1/1 | 1 |
| Score | Role | Decision / admitted rows | Admitted logs | Coverage [95% CI] | Helpful / false / risk |
|---|---|---|---|---|---|
| Basin spread | Calibration | never-admit / 0 of 90 | 0 of 30 | [ ] | n/a / n/a / n/a |
| Basin spread | Held-out | never-admit / 0 of 90 | 0 of 30 | [ ] | n/a / n/a / n/a |
| Legacy cost drop | Calibration | never-admit / 0 of 90 | 0 of 30 | [ ] | n/a / n/a / n/a |
| Legacy cost drop | Held-out | never-admit / 0 of 90 | 0 of 30 | [ ] | n/a / n/a / n/a |
| Parameter | Value |
|---|---|
| Candidate LiDAR points (“edge”) | Range-discontinuity points: adjacent, within the same scan ring in azimuth order, to a range jump m; range window – m; capped at 2,500 points, subsampled uniformly at random if exceeded. |
| Random-point control | Same range window and point budget as the edge set, drawn uniformly at random with no discontinuity selection. |
| Image cost surface | Canny edge map (OpenCV, thresholds 50/150) on grayscale, inverted, then an L2 distance transform with a mask. |
| Depth / occlusion filtering | A candidate point is kept only if it is in front of the camera ( m) and within a normalized radius (bounds AV2’s radial distortion polynomial to its well-behaved range). If fewer than 20 points survive filtering, the cost function returns a fixed penalty (200) instead of a projection-based value. |
| Dynamic-object handling | None: no separate moving-object filter is applied beyond the depth/radius and range-window checks above. This is disclosed as a limitation, not assumed away. |
| Optimizer | scipy.optimize.minimize , Powell method, maxiter=60 , xtol=1e-4 , ftol=1e-4 . |
| Role | Exclusive use |
|---|---|
| Development | Baseline, Oracle, zero-injection, and online-motion smoke tests only; no reported threshold or final metric. |
| Calibration | Fix baseline configuration and ; choose the guard operating point once. |
| Final | Run all arms and report intervals; never tune parameters or replace a failed baseline. |
| External transfer | Deferred unless the Argoverse 2 system audit clears its locked safety and benefit gates. |
| Arm | Extrinsic policy |
|---|---|
| Raw | Always use injected . |
| Oracle | Always use true ; upper reference and adapter health check. |
| Always-correct | Always inject ; exposes recursive amplification of a bad correction. |
| Guarded | Admit only temporal basin-consistent corrections; otherwise fall back to Raw. |
| Criterion | Required outcome |
|---|---|
| Benefit | Guarded improves Raw on ATE or RPE in the tier-blind final mixture; paired-log 95% CI excludes zero. |
| Safety | Guarded is non-inferior to Raw on every predeclared safety metric, including zero injection. |
| Mechanism | Always-correct versus Guarded is reported. If it is worse, this supports recursive-risk mitigation; if not, no necessity claim. |
| Failure handling | Any failed criterion is a negative result. Parameters, thresholds, and final logs are not recycled for a new attempt. |
| Role | (m) | Observed | Null mean | Observed–null [95% CI] | logs |
|---|---|---|---|---|---|
| Formal 27-log | 0.1 | 6.523 | 8.944 | [ ] | 11/27 |
| Formal 27-log | 1.0 | 4.990 | 6.314 | [ ] | 7/27 |
| Formal 27-log | 10.0 | 4.965 | 6.259 | [ ] | 7/27 |
| Exploratory 60-log | 0.1 | 8.069 | 10.164 | [ ] | 21/60 |
| Exploratory 60-log | 1.0 | 5.734 | 7.540 | [ ] | 22/60 |
| Exploratory 60-log | 10.0 | 5.686 | 7.482 | [ ] | 20/60 |