Queryable Gaussian View Index: Text Retrieval and Selective Camera Localization
Organizations: School of Computer Science, Northwestern Polytechnical University Xi’an, China · CEMSE, King Abdullah University of Science and Technology Thuwal, Saudi Arabia
Abstract
We present Queryable Gaussian View Index (QGVI) for querying existing Gaussian scene assets and their associated calibrated reference data. Text-view retrieval and selective RGB camera localization return poses in the asset's coordinate frame. QGVI-T turns Gaussian-rendered views into a language-searchable index of scene locations, binding structured captions to known camera poses and place identifiers. Semantic and lexical retrieval select views from distinct places without online rendering, VLM calls, or a scene-specific pose network. The image route, QGVI-I, filters LoGS-based geometric estimates using distinct inlier query keypoints. It omits photometric refinement and preserves accepted pose coordinates. On nine development scenes, QGVI-T achieves 58.3% Top-1 and 78.3% Top-5 joint relevance-and-validity success, compared with 26.1% and 34.6% for our CityLoC reimplementation with refinement. QGVI-I achieves 97.3% scene-mean pose success at 1 m/2 degrees, with pooled false-acceptance and false-rejection rates of 0.707% and 1.142%. A paired cached evaluation retains all 1,775 initializer poses correct at this threshold. Recorded median text query latency is 0.020 s. Matched controls show competitive BM25 retrieval and slightly lower false acceptance for thresholded full LoGS; grid augmentation suggests modest gains on novel-view-derived descriptions. We report model-judged semantic success separately from source-camera recovery, with source-verified or estimated scales for geometric evaluation.
Figures & tables
| Method | Coverage (%) | Answer precision (%) | Correct returns (%) | Wrong returns (%) | Net utility | |
|---|---|---|---|---|---|---|
| Text queries (including optional verification) | ||||||
| CityLoC w/o refinement | 540 | 100.00 | 25.00 | 25.00 | 75.00 | -50.00 |
| CityLoC with refinement | 540 | 80.19 | 32.56 | 26.11 | 54.07 | -27.96 |
| QGVI-T | 180 | 100.00 | 58.33 | 58.33 | 41.67 | 16.67 |
| QGVI-T + verification | 180 | 75.00 | 88.15 | 66.11 | 8.89 | 57.22 |
| RGB queries (50% out-of-map) | ||||||
| Method / support threshold | Pose | FAR | FRR |
|---|---|---|---|
| Initializer: count | 96.52 | 1.305 | 1.033 |
| Initializer: ratio | 96.36 | 79.445 | 0.761 |
| QGVI-I: distinct | 96.52 | 0.707 | 1.142 |
| LoGS: pairs | 96.52 | 1.251 | 1.142 |
| LoGS: distinct | 96.52 | 0.544 | 1.142 |
| SplatHLoc: native ( ) | 61.17 | 99.347 | 0.272 |
| Retrieval | Top-1 | Top-5 |
|---|---|---|
| CLIP | 39.4–54.4 | 62.2–80.6 |
| BM25 | 62.8 –78.9 | 77.2 –92.2 |
| MiniLM | 56.7–73.9 | 70.6–90.0 |
| QGVI-T (fusion) | 61.1–77.8 | 73.9–91.7 |
| Description source | Top-1 | Top-5 | |||
|---|---|---|---|---|---|
| O | O+G | O | O+G | ||
| All descriptions | 180 | 57.8 | 60.0 | 73.3 | 74.4 |
| Original-image source | 74 | 52.7 | 52.7 | 71.6 | 70.3 |
| Novel-render source | 106 | 61.3 | 65.1 | 74.5 | 77.4 |
| Method | Query time (s) Median / p95 |
|---|---|
| CityLoC w/o refinement | 3.376 / 8.389 |
| CityLoC with refinement | 171.720 / 429.821 |
| QGVI-T | 0.020 / 0.063 |
| LoGS | 0.436 / 2.293 |
| SplatHLoc | 0.510 / 0.847 |
| QGVI-I | 0.240 / 0.450 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Metric (%) | Block Small | Block A | NYC | SF | Citysample | Colosseum | Elvenruin | Park | Road | Scene mean |
|---|---|---|---|---|---|---|---|---|---|---|---|
| In-map queries | 139 | 602 | 180 | 174 | 142 | 126 | 118 | 228 | 130 | — | |
| Out-of-map queries | 139 | 602 | 180 | 174 | 142 | 126 | 118 | 228 | 130 | — | |
| Scale | m / native unit | 25 | 25 | 1 | 1 | 24.777 † | 12.287 † | 14.135 † | 1.040 † | 1.339 † | — |
| Gaussian-depth pipeline (v3) | Pose 1 m/2 ∘ | 63.3 | 60.6 | 100.0 | 100.0 | 88.0 | 64.3 | 94.9 | 82.9 | 76.2 | 81.1 |
| Pose 5 m/5 ∘ | 64.0 | 62.0 | 100.0 | 100.0 | 95.8 | 64.3 | 97.5 | 84.2 | 78.5 | 82.9 | |
| False accept | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| Scene | Reference bank | Views | Pose success 1 m/2 ∘ (%) | Pose success 5 m/5 ∘ (%) | Out-of-map accept (%) | Query time (s) Median / 95th percentile |
|---|---|---|---|---|---|---|
| Block Small | Baseline | 1,886 | 13.7 | 14.4 | 0.0 | 0.117 / 0.588 |
| Equal-size replacement | 1,886 | 15.8 | 16.5 | 0.0 | 0.109 / 0.587 | |
| Full expansion | 1,945 | 15.8 | 16.5 | 0.0 | 0.109 / 0.591 | |
| Block A | Baseline | 13,094 | 1.5 | 1.5 | 0.0 | 0.125 / 0.221 |
| Equal-size replacement | 13,094 | 1.5 | 1.5 | 0.0 | 0.124 / 0.209 | |
| Full expansion | 14,347 | 1.5 | 1.5 | 0.0 | 0.126 / 0.214 |
| Scene | Queries | Median (ms) | 95th percentile (ms) |
|---|---|---|---|
| Block Small | 278 | 53.184 | 72.137 |
| Block A | 1,204 | 101.922 | 122.144 |
| NYC | 360 | 50.939 | 73.183 |
| SF | 348 | 59.176 | 86.112 |
| Citysample | 284 | 54.040 | 59.646 |
| Colosseum | 252 | 44.233 | 59.522 |
| Method | P1 1 m/2 ∘ | R5 1 m/2 ∘ | P1 5 m/5 ∘ | R5 5 m/5 ∘ | Mean scene median (m/ ∘ ) | Pose coverage (%) |
|---|---|---|---|---|---|---|
| All nine scenes, 180 queries | ||||||
| CityLoC w/o refinement | 0.00 | 0.00 | 0.19 | 0.19 | 86.30 / 69.99 | 100.00 |
| CityLoC with refinement | 0.00 | 0.00 | 0.37 | 0.56 | 89.16 / 64.98 | 80.19 |
| QGVI-T | 0.00 | 0.00 | 0.56 | 3.33 | 64.66 / 51.64 | 100.00 |
| QGVI-T + verification | 0.00 | 0.00 | 0.56 | 4.44 | 47.08 / 55.26 | 75.00 |
| Four scenes with source-verified scales, 80 queries | ||||||
| Method | Coverage | Top-1 success | Top-5 success | Net utility | ||||
|---|---|---|---|---|---|---|---|---|
| Kimi | GPT | Kimi | GPT | Kimi | GPT | |||
| CityLoC w/o refinement | 540 | 100.0 | ||||||
| CityLoC with refinement | 540 | 80.2 | ||||||
| QGVI-T | 180 | 100.0 | ||||||
| QGVI-T + verification | 180 | 75.0 | ||||||
| Scene | CityLoC w/o refinement | CityLoC with refinement | QGVI-T | QGVI-T + verification |
|---|---|---|---|---|
| Block Small | 4.776 / 5.915 | 145.724 / 609.644 | 0.021 / 0.029 | 15.457 / 16.844 |
| Block A | 8.244 / 10.336 | 182.853 / 700.906 | 0.062 / 0.086 | 15.648 / 17.970 |
| NYC | 3.297 / 3.693 | 198.926 / 327.830 | 0.018 / 0.022 | 15.462 / 16.845 |
| SF | 2.717 / 3.660 | 177.899 / 347.767 | 0.020 / 0.024 | 15.334 / 16.155 |
| Citysample | 3.113 / 3.772 | 85.713 / 239.108 | 0.021 / 0.024 | 15.380 / 16.164 |
| Colosseum | 2.559 / 3.081 | 170.950 / 226.452 | 0.012 / 0.015 | 15.836 / 16.915 |
| Scene | LoGS | SplatHLoc | QGVI-I |
|---|---|---|---|
| Block Small | 0.487 / 0.789 | 0.529 / 0.630 | 0.286 / 0.342 |
| Block A | 0.425 / 0.722 | 0.527 / 0.544 | 0.329 / 0.647 |
| NYC | 1.065 / 2.732 | 0.399 / 0.877 | 0.193 / 0.246 |
| SF | 0.365 / 0.776 | 0.107 / 0.857 | 0.232 / 0.300 |
| Citysample | 0.343 / 1.424 | 0.543 / 0.649 | 0.198 / 0.262 |
| Colosseum | 0.318 / 0.366 | 0.118 / 0.613 | 0.227 / 0.287 |
| Method | Metric (%) | Block Small | Block A | NYC | SF | Citysample | Colosseum | Elvenruin | Park | Road | Scene mean |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Queries | 20 | 20 | 20 | 20 | 20 | 20 | 20 | 20 | 20 | — | |
| CityLoC [ 26 ] without refinement | Top-1 success | 21.7 | 15.0 | 21.7 | 43.3 | 30.0 | 33.3 | 20.0 | 20.0 | 20.0 | 25.0 |
| Top-5 success | 25.0 | 21.7 | 21.7 | 45.0 | 33.3 | 40.0 | 23.3 | 35.0 | 31.7 | 30.7 | |
| No result | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | |
| CityLoC [ 26 ] with refinement | Top-1 success | 21.7 | 15.0 | 16.7 | 48.3 | 35.0 | 38.3 | 20.0 | 16.7 | 23.3 | 26.1 |
| Top-5 success | 25.0 | 18.3 | 30.0 | 56.7 | 43.3 | 50.0 | 25.0 | 31.7 | 31.7 | 34.6 |
| Scene | CLIP | BM25 | MiniLM | QGVI-T (hybrid) | ||||
|---|---|---|---|---|---|---|---|---|
| Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | |
| Block Small | ||||||||
| Block A | ||||||||
| NYC | ||||||||
| SF | ||||||||
| Citysample | ||||||||
| Scene | Original images | Grid renders | Original + grid | |||
|---|---|---|---|---|---|---|
| Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | |
| Block Small | ||||||
| Block A | ||||||
| NYC | ||||||
| SF | ||||||
| City Sample | ||||||
| Cohort | Candidate bank | Images | Top-1 (%) | Top-5 (%) | Top-5 vs O | Top-5 vs R |
|---|---|---|---|---|---|---|
| Block A | Same-pose renders | 5,138 | ||||
| Block A | Expanded renders | 18,232 | ||||
| SF | Same-pose renders | 666 | ||||
| SF | Expanded renders | 3,549 | ||||
| Pooled | Same-pose renders | 5,804 | ||||
| Pooled | Expanded renders | 21,781 |
| Scene | Policy | Nominal (%) | Actual (%) | Images | Mean Top-1 bounds (%) | Mean Top-5 bounds (%) | Top-5 seed range (%) | Top-5 |
|---|---|---|---|---|---|---|---|---|
| Block A | Replace | 0 | 0.0 | 5,138 | ||||
| Block A | Replace | 25 | 25.0 | 5,138 | ||||
| Block A | Replace | 50 | 50.0 | 5,138 | ||||
| Block A | Replace | 75 | 75.0 | 5,138 | ||||
| Block A | Grow | 25 | 25.0 | 8,412 | ||||
| Block A | Grow | 50 | 50.0 | 11,685 |
| Source ( ) | Metric | W/L/S/U | (pp) |
|---|---|---|---|
| Original (74) | Top-1 | 1/1/67/5 | |
| Original (74) | Top-5 | 0/0/64/10 | |
| Render (106) | Top-1 | 4/0/89/13 | |
| Render (106) | Top-5 | 1/1/82/22 |
| Acceptance rule | Threshold | FAR | FRR | Pose success ( ) | Precision | ||
|---|---|---|---|---|---|---|---|
| ( ) | ( ) | ( ) | |||||
| Fixed LoGS initializer poses (no photometric refinement) | |||||||
| Return every pose | — | 100.000 | 0.000 | 96.52 | 96.68 | 48.26 | -3.48 |
| PnP success only | — | 99.674 | 0.054 | 96.52 | 96.68 | 48.35 | -3.29 |
| Raw inlier count | 40 | 1.305 | 1.033 | 96.52 | 96.68 | 96.26 | 46.38 |
| Raw match inlier ratio | 0.1 | 79.445 | 0.761 | 96.36 | 96.47 | 53.93 | 7.01 |
| Scene | Pose success ( ) | FRR ( ) | FAR ( ) | Precision ( ) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Q | L | Q | L | Q | L | |||||
| Block Small | 139 | 90.65 | 92.09 | 3.597 | 0.000 | 0.000 | 94.03 | 94.03 | 42.45 | 42.45 |
| Block A | 602 | 93.36 | 93.52 | 2.492 | 0.332 | 0.332 | 95.42 | 95.42 | 44.44 | 44.44 |
| NYC | 180 | 100.00 | 100.00 | 0.000 | 1.111 | 0.556 | 98.90 | 99.45 | 49.44 | 49.72 |
| SF | 174 | 100.00 | 100.00 | 0.000 | 4.023 | 2.874 | 96.13 | 97.21 | 47.99 | 48.56 |
| City Sample | 142 | 100.00 | 100.00 | 0.000 | 1.408 | 1.408 | 98.61 | 98.61 | 49.30 | 49.30 |
| Metric | QGVI-I w/o gate | QGVI-I |
|---|---|---|
| Accepted outputs ( ) | 3,678 | 1,831 |
| False accepts / out-of-map | 1,839/1,839 | 13/1,839 |
| False accept rate | 100.000 | 0.707 |
| False rejects / in-map | 0/1,839 | 21/1,839 |
| False reject rate | 0.000 | 1.142 |
| Pose success, 1 m/2 ∘ | 96.52 | 96.52 |
| Retrieval | Visual verifier | Relevant Top-5 success (%) | |
|---|---|---|---|
| CLIP | No | 176 | 68.2–97.2 |
| CLIP | Yes | 176 | 76.1–81.8 |
| Captions (QGVI-T) | No | 176 | 80.7– 99.4 |
| Captions | Yes | 176 | 84.1 –88.1 |
| Scene / query type | CLIP + verifier Top-5 (%) | QGVI-T + verification Top-5 (%) | |
|---|---|---|---|
| Block Small | 20 | 85.0 | 85.0 |
| Block A | 20 | 80.0– 85.0 | 85.0 |
| NYC | 20 | 90.0 | 90.0 |
| SF | 20 | 80.0–95.0 | 85.0 – 100.0 |
| Citysample | 20 | 65.0–70.0 | 80.0 |
| Colosseum | 16 | 75.0–87.5 | 87.5 – 100.0 |
Camera Pose Estimation
Camera pose estimation recovers a camera’s position and orientation from images, visual features, scene geometry, and sometimes sensor measurements.
See all 264 papers