cs.ROOct 27, 2026
Save3D Occupancy-Language Fields from mmWave Radar
Organizations: University of Pennsylvania, Philadelphia, PA, 19104, USA.
Abstract
Radar systems can recover scene geometry and recognize predefined categories, but these outputs do not provide a persistent spatial representation for open-ended language queries. Such queries require linking observations from different viewpoints to the same physical locations. We present TextRadar, a framework that constructs a scene-level 3D occupancy-language field from panoramic radar observations. The field combines occupancy with scale-dependent language features at surface locations, allowing the reconstructed scene to be queried with new descriptions. A single-frame radar distillation model predicts aligned ranges and language features from each radar observation. Using radar-estimated poses, we register these predictions to reconstruct the field from radar alone in a new scene. Experiments on approximately 20k synchronized observations across seven buildings demonstrate scene-level natural-language querying and prompt-free semantic visualization. Analysis of camera-to-radar concept transferability shows stronger agreement for structures, materials, and human poses than for colors and printed text.
Figures & tables
Figure 1: TextRadar constructs a scene-level 3D occupancy-language field from radar observations. A robot collects panoramic mmWave measurements along a trajectory (left). Surface geometry and language features predicted from these measurements are integrated in shared 3D coordinates (center). The right panels illustrate natural-language querying, prompt-free semantic visualization, and camera-to-radar concept transferability. The query “Things to sit” highlights a bench, while the feature visualization reveals semantic organization without specifying a prompt. Comparisons of radar-predicted and camera-derived features show stronger agreement for structures, objects, materials, and human poses than for colors and printed text.
Figure 2: Scene-level natural-language querying of large building structures. The reconstructed geometry is colored according to the cosine similarity between the scene-level feature at each surface point and the CLIP text embedding of the query. Warmer colors indicate higher semantic relevance. The queries (a) “Ceiling” and (b) “Floor” activate complementary upper and lower surfaces across multiple corridors and room regions. The spatially coherent responses over distant parts of the environment show how the persistent occupancy-language field associates related structures across a common 3D coordinate system.
Figure 3: Scene-level natural-language querying of small structures and objects. The field localizes architectural structures, including (a) “Stairs” and (b) “Windows,” as well as compact objects, including (c) “Trash bin” and (d) “Human.” The RGB images provide representative observations of the queried regions, with dashed boxes indicating the corresponding targets. For categories with multiple instances, such as stairs and windows, the system simultaneously highlights geometrically separated regions throughout the scene.
Figure 4: Geometry and natural-language queries. Each example shows, from top to bottom, the rendered panoramic RGB reference, the range image predicted from the radar measurement, the query-relevancy map obtained from the radar-predicted CLIP-aligned features, and the corresponding map obtained from the camera-derived reference features. Warmer colors indicate higher cosine similarity to the text query. The queries cover materials, including (a) “Glass,” (b) “Metal,” and (c) “Marble”; objects, including (d) “Trash bin,” (e) “Chairs,” and (f) “Human”; and structures, including (g) “Stairs,” (h) “Floor,” and (i) “Ceiling.” The radar-predicted maps localize both compact objects and extended structural regions, remain spatially consistent with the recovered geometry, and broadly agree with the camera-derived reference maps.
Figure 5: Natural-language querying of high-level concepts. The prompts describe human configuration, affordances, and functional regions: (a) “Standing” localizes a standing person, (b) “Going down” highlights a descending staircase, (c) “Things to sit” retrieves a bench and other seating, (d) “Taking a rest” identifies a shared seating area, (e) “Entrance to other places” highlights doorways and passages, and (f) “Working region” identifies the occupied workspace in a laboratory office. These examples show responses to contextual prompts beyond direct object and material names.
Figure 6: Prompt-free semantic visualization of general indoor structures. For each example, the rows from top to bottom present the rendered panoramic RGB image, the predicted range image, the three-component PCA projection of the radar-predicted CLIP features, and the corresponding camera-derived reference projection. The predicted and reference features are projected using a shared PCA basis, and the three principal components are mapped to RGB channels. Similar colors therefore indicate nearby positions in the projected semantic feature space, rather than predefined semantic classes. Large structural regions, including ceilings, floors, walls, corridor openings, and wall-mounted structures, form spatially coherent groups in the radar-predicted visualization and broadly agree with the camera-derived reference.
| Pairwise | Region | Instance-level | |||
| Category | Evaluation entity | # prompts | discrimination | AUROC | correlation |
| (chance 0.50) | (chance 0.50) | (null 0.00) | |||
| Objects / structures | whole frame | 16 | 0.825 | 0.859 | 0.885 |
| Materials | one object class | 6 | 0.856 | 0.694 | 0.542 |
| Human actions | person | 6 | 0.755 | 0.704 | 0.552 |
| Colors | one object class | 9 | 0.669 | 0.592 | 0.273 |
Table 1: Category-wise radar-camera concept agreement. Camera-derived CLIP relevancy is the reference signal, and every property is evaluated within the entity that carries it. Values are means over the seven held-out buildings and the complete prompt bank.
Figure 7: Prompt-free semantic visualization of structures and objects. The examples examine (a) stairs, (b) humans, and (c) chairs, with two scenes shown for each concept. The dashed boxes in the rendered RGB image indicate the structures or objects of interest. A shared PCA projection is used for each prediction-reference comparison, allowing their color organization to be compared directly. Stairs, human silhouettes, and individual chairs emerge as coherent regions whose boundaries follow the corresponding geometry, despite some differences in their sizes, shapes, and spatial extents.
Figure 8: Prompt-free semantic visualization of material-related regions. The examples show (a) wood doors, (b) a glass window, and (c) a metal cabinet. The highlighted material regions occupy distinct portions of the projected feature space and therefore appear with colors different from their surrounding walls and structures. The agreement between the radar-predicted and camera-derived PCA images indicates material-related information being retained beyond range geometry alone.
| Category | B1 | B2 | B3 | B4 | B5 | B6 | B7 | Mean |
|---|---|---|---|---|---|---|---|---|
| Pairwise discrimination (chance 0.50) | ||||||||
| Objects / structures | 0.819 | 0.772 | 0.776 | 0.840 | 0.811 | 0.900 | 0.860 | 0.825 |
| Materials | 0.860 | 0.893 | 0.813 | 0.858 | 0.770 | 0.929 | 0.867 | 0.856 |
| Human actions | 0.741 | 0.741 | 0.708 | 0.777 | 0.733 | 0.787 | 0.799 | 0.755 |
| Colors | 0.636 | 0.609 | 0.563 | 0.651 | 0.643 | 0.790 | 0.793 | 0.669 |
| Text | 0.439 | 0.583 | 0.565 | 0.510 | 0.503 | 0.730 | 0.726 | 0.579 |
Table 2: Per-building radar-camera concept agreement. Each building is the held-out test split in the leave-one-building-out protocol.
Figure 9: Radar-camera feature disagreement. The top row highlights concepts dominated by visual content: (a) televisions and monitors, (b) information signs, and (c) exhibition boards. The bottom row highlights appearance-dependent concepts: (d) ribbed texture, (e) yellow color, and (f) roughness and grid patterns. Dashed boxes mark the relevant regions in the RGB observations. While the camera-derived features clearly distinguish these regions by displayed content, color, or fine surface appearance, the radar-predicted features often assign them similar embeddings and thus fail to distinguish them.
| Range estimation | Radar-camera cosine similarity (%) | |||||||
| Building | Median (cm) | Mean (cm) | PSNR | Scale 1 | Scale 2 | Scale 3 | Scale 4 | Scale 5 |
| B1 | 5.49 | 17.92 | 26.16 | 92.13 | 92.62 | 92.56 | 91.86 | 91.12 |
| B2 | 5.00 | 19.77 | 26.36 | 90.76 | 91.56 | 91.71 | 91.07 | 90.19 |
| B3 | 5.50 | 24.04 | 23.16 | 88.59 | 89.20 | 88.06 | 88.44 | 87.62 |
| B4 | 4.03 | 15.16 | 28.93 | 93.42 | 93.67 | 93.57 | 92.93 | 91.94 |
| B5 | 4.49 | 14.64 | 29.04 | 91.06 | 91.39 | 91.14 | 90.15 | 89.03 |
Table 3: Per-building geometry and multiscale CLIP feature prediction in held-out environments. Each building is evaluated as the held-out test split in the leave-one-building-out protocol.
| Model | Range error (cm) | Range image PSNR (dB) | Radar-camera cosine similarity (%) |
|---|---|---|---|
| DPT with shared decoder | 16.29 | 27.44 | 88.36 |
| UNet with separate decoder | 16.35 | 27.40 | 90.08 |
| UNet with shared decoder (ours) | 16.38 | 27.38 | 91.92 |
Table 4: Model and decoder ablation for range and CLIP feature prediction. We compare a transformer-based dense prediction model, a CNN with separate task decoders, and our CNN with a shared decoder. Metrics are averaged over all held-out buildings and, for cosine similarity, all five scales.
Figure 10: Hardware platform for data collection. The mobile robot carries a rotating mmWave MIMO radar, a 3D LiDAR, and a front-facing camera. A stepper motor rotates the radar, while the battery and WiFi router rotate with the radar to prevent cable twisting and are placed behind it for weight balancing. Radar commands are sent through the wireless router. (a) Front view. (b) Rear view.
Figure 11: Example scenarios in our dataset. The dataset spans common indoor environments, including hallways, lounges, offices, classrooms, and open floor areas. These examples are captured by the front-facing camera on the robot and illustrate the visual context used to generate language supervision.
Figure 12: Radar-camera feature disagreement around glass. Each column presents one example, and the rows show the rendered panoramic RGB reference, the radar-predicted range image, and the pixelwise cosine similarity between radar-predicted and camera-derived CLIP features. Dashed orange boxes indicate glass windows. Warmer colors represent stronger feature agreement, and cooler colors represent weaker agreement. Similarity decreases around the indicated glass regions, where the two sensors observe different scene content.
Figure 13: Overview of the TextRadar framework. A rotating mmWave radar provides high-resolution panoramic RF measurements, while a front-facing camera supplies multiscale language supervision during data collection. Building on these inputs, TextRadar jointly learns geometry and language-aligned embedding from RF measurements and builds a scene-level 3D occupancy-language field, which supports several tasks.
| Category | Prompts |
|---|---|
| Objects / structures (16) | a flat interior wall; the floor of the room; the ceiling overhead; a door in the hallway; a trash bin on the floor; a person standing in the room; a glass window or panel; a table; a chair; a staircase with steps; a vertical pillar or column; a handrail or railing; a television screen on the wall; a window on the wall; a carpeted floor; an elevator door |
| Colors (9) | a {red, blue, green, yellow, orange, black, white, gray, pink} colored object |
| Materials (6) | a wooden surface; a metal surface; a marble surface; an exposed aggregate concrete surface; a plastic surface; a ceramic surface |
| Human actions (6) | a person standing upright; a person sitting down; a person lying on the floor; a person raising their arms; a person bending down; a person crouching |
| Text (5) | a digital display showing numbers; printed letters on a sign; the number 7; the letter A; a line of printed words |
Table 5: Prompt bank for the camera-to-radar concept transferability analysis. The bank contains 42 queries across the five semantic categories used in the evaluation.