A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans approximately 11.8M video frames and approximately 730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at https://dreamvu.ai/prism
Figures & tables
Figure 1: PRISM captures multi-view retail video from four synchronized modalities - egocentric, exocentric, 360° panoramic and depth - and structures 270K video SFT samples across 20+ task types organized into four capability dimensions. All capabilities feed into a model-agnostic fine-tuning format compatible with any VLM or VLA, producing embodied agents for real-world retail deployment.
Figure 2: PRISM pipeline overview. The overview of the PRISM pipeline is shown here. PRISM is built on egocentric and exocentric videos from real-world retail stores. Four annotation strategies - metadata extraction, LLM generation (Gemini 2.5 Flash), depth analysis (DepthCrafter), and self-supervised transformations - produce twenty tasks (full list of tasks is mentioned in section section 3 ) across four capability domains, totaling 270K instruction-tuning samples. Solid boxes denote egocentric tasks; dashed boxes denote exocentric tasks ( exo ). Samples from PRISM are used for finetuning Cosmos-Reason2-2B via BF16 LoRA.
Domain
Cosmos-Reason1 (base model)
PRISM (ours)
Source datasets
SFT samples
Tasks
SFT samples
Embodied
BridgeV2, AgiBot ∗ ,
∼ 1,720K
ER-1–ER-9 (9 tasks)
93,757
Reasoning
HoloAssist, RoboVQA
(general manip.; no retail SFT)
incl. multi-actor ‡ , cross-view ‡
Common
Physical CS
∼ 1,800K †
CS-U-1/2, CS-R-1–4
47,778
Sense +
(curated video + MCQ)
(general scenes)
SP-1, SP-2 ‡ (360°)
11,862
Spatial
incl. depth-grounded, exo spatial ‡
Table 1: PRISM vs. Cosmos-Reason1 training data by capability domain. Cosmos-Reason1 NVIDIA (2025b) SFT counts are taken from the publicly released HuggingFace datasets. While AgiBot World includes retail scenes as one of five deployment domains ( ∗ ), its SFT samples target general manipulation skills and are not structured as retail-domain perceptual QA; no dataset contributing to Cosmos-Reason1 provides retail-targeted VLM SFT. PRISM contributes 270K retail-specific SFT pairs spanning all three domains. † Cosmos Physical Common Sense covers generic scenes; no retail or robot-deployment content is included. ‡ Novel tasks entirely absent from Cosmos-Reason1 training. ∗ AgiBot World includes a retail deployment domain ( ∼ 1 of 5 domains, supermarket shelf-picking and restocking tasks), but contributes no retail-targeted VLM SFT to Cosmos-Reason1 training.
Dataset
Scale
#Tasks
Viewpoint
Domain
CoT
SFT
Ego4D Grauman and others (2022)
3,670 h
5
Ego
General
×
×
Ego-Exo4D Grauman and others (2024)
1,286 h
4
Ego + Exo
Skilled act.
×
×
RoboVQA Sermanet et al. (2023)
829K pairs
6
Exo
Office robot
×
✓
SariBench Gajo et al. (2025)
100 demos
3
Ego (sim.)
Retail (sim.)
×
×
PRISM (ours)
270K samples
20
Ego+Exo+360°
Retail
✓
✓
Table 2: Comparison with related embodied video datasets. PRISM is the only dataset that simultaneously covers retail deployment, all three viewpoint modalities (ego, fixed exo, 360° exo), explicit chain-of-thought supervision, and a VLM SFT-ready format. ✓ = yes; × = no.
Figure 3: PRISM capability probe examples (Part 1 of 3). Each row shows a representative video frame alongside the full question and model answer. CoT tasks include the chain-of-thought in ⟨ think ⟩ tags (shown in italics) before the final answer.
Figure 4: PRISM capability probe examples (Part 2 of 3). Continued from above.
Figure 5: PRISM capability probe examples (Part 3 of 3). CS-R-4 and SP tasks test spatial understanding. IP-1 CoT variants demonstrate physics-grounded reasoning about temporal direction. IP-2 evaluates object permanence. MCQ Overlay converts open-ended tasks into four-choice format.
Task
Domain
View
Format
Allocated
% of Total
Annotation Method
ER-1: Next Subtask Prediction
ER
Ego
Und.+MCQ
25,000
9.2%
Gemini Robotics ER 1.5
ER-2: Task Completion Verification
ER
Ego
Und
3,000
1.1%
Gemini Robotics ER 1.5
ER-3: Goal-Conditioned Action Reasoning
ER
Ego
CoT
8,000
3.0%
LLM (Gemini Flash)
ER-4: Exo-to-Ego Activity Matching
ER
Ego+Exo
Und.
7,000
2.6%
LLM (Gemini Flash) + Video
ER-5: Hand Interaction Recognition
ER
Ego
Und.+MCQ
15,000
5.5%
Gemini Robotics ER 1.5
ER-6: Atomic Action Recognition
ER
Exo
Und.
20,000
7.4%
Gemini Robotics ER 1.5
Table 3: PRISM dataset overview. All 20+ capability probes organized by knowledge domain, viewpoint, SFT format, allocated sample counts, and annotation method. Und. = open-ended understanding; CoT = chain-of-thought reasoning with ⟨ think ⟩ tags; MCQ = multiple-choice (A/B/C/D). Sample allocations follow sqrt-proportional scaling with no single task exceeding 9.5% of the total mix.
Metric
Value
Total samples
270K
Tasks (+ 2 eval-only)
20+
Video frames (4 fps)
∼ 11.8M
Total tokens
∼ 730M (703M visual + 27M text)
Open-ended / CoT / MCQ
58.6% / 30.4% / 9.2%
Domain: ER / CS / SP / IP / MCQ
34.7 / 18.3 / 4.4 / 33.5 / 9.2%
Table 4: PRISM scale statistics. Token counts at 4 fps with Qwen3VL tokenization.
Domain-TaskID: Task
View
Baseline (%)
PRISM (%)
Δ (%)
Error Ratio
ER-1: Next Subtask Prediction
Ego
30.8
76.8
+46.0
0.34
ER-2: Task Completion
Ego
53.6
87.6
+34.0
0.27
ER-3: Action Reasoning
Ego
32.4
95.2
+62.8
0.07
ER-4: Cross-View Matching ⋆
Ego+Exo
42.9
91.0
+48.1
0.16
ER-5: Hand Interaction
Ego
61.6
95.6
+34.0
0.11
ER-6: Atomic Action ⋆
Exo
54.0
84.8
+30.8
0.33
Table 5: Performance Comparison between Baseline (Zero-Shot Cosmos-Reason2-2B) and PRISM (Our model fine-tuned on the full 270K training corpus) on all evaluation tasks. Best score per task in bold . The Error Ratio shows the ratio of PRISM fine-tuned error rate to the baseline error rate.
Figure 6: Performance scaling with training data. MCQ accuracy by capability domain as a function of training progress. All domains improve rapidly in early training (0–40%), with ER showing the largest absolute gain (+31.5pp). Performance continues to increase at a diminishing rate, with the full 270K dataset achieving the best overall accuracy of 88.9%. CS starts highest due to strong baseline scene understanding; IP shows steady improvement throughout training.
Example 1
Context: Goal: “Perform Grocery Shopping”. Sub-goal: “Navigate Produce Section”. Scene: Produce Section. Completed: “Aisle Navigation (Fruits and Vegetables).”
Q: What is the next subtask the person will perform?
A: The next subtask is: “Aisle Navigation (Dry Goods)”. Walk forward into an aisle.
Q: What is the next subtask the person will perform?
Table 12
Example 1
Q: Can you determine if the task “Navigate Aisle while Reviewing List” has been completed?
A: No, the task has not been completed yet. The person walks from the produce section towards the refrigerated freezers, checking their phone.
Example 2
Q: Based on what you see, has the person finished “Wait for Checkout”?
A: The task is not yet complete. The person turns back towards the checkout counter, resuming their wait in line.
Example 3
Table 13
Example 1
Q: Based on the person’s current task, where should they navigate to next in the store?
A: ⟨ think ⟩ The person has just finished inspecting the Heritage Platinum Basmati Rice. Their current sub-goal is “Examine Product Details,” which they have completed. The next logical step is to find another product to compare. ⟨ /think ⟩ Continue down the current aisle to explore other rice options or adjacent product categories.
Example 2
Q: Why is the person performing this action in the context of their shopping goal?
A: ⟨ think ⟩ The person is reaching for a spice packet while their shopping list is visible on their phone. This indicates they are actively checking off items from a prepared list. ⟨ /think ⟩ The person is selecting a specific spice that matches an item on their shopping list.
Table 14
Example 1
Q: What activity is the person performing in this exocentric view?
A: The worker is scanning shelf labels and restocking items in the beverage aisle. They methodically check each section before placing products from a cart onto the shelves.
Example 2
Q: Describe the activity visible from this overhead camera perspective.
A: A customer is browsing the produce section, picking up and inspecting individual vegetables before placing selected items into a blue shopping basket.
Table 15
Example 1
Q: Describe what the person is doing with their hands in this clip.
A: Both hands are holding an item. The person pushes the cart down the aisle, passing shelves filled with rice bags and other packaged goods.
Example 2
Q: Describe what the person is doing with their hands in this clip.
A: The left hand is holding an item. The right hand is resting and not engaged. The person stops near the vegetable displays and observes another customer selecting carrots from a crate.
Table 16
Example 1
Q: Based on the video, what action is the person currently performing?
A: The action being performed by the man is: reaching toward basket (target: blue shopping basket). Extends right arm towards the blue basket.
Example 2
Q: Describe the atomic action visible in this clip.
A: The person is performing: walking forward (direction: along aisle). Observable cues: full body translation, upright posture, consistent forward gait.
Table 17
Example 1
Q: What is the likely reason this person is doing what they’re doing in this scene?
A: ⟨ think ⟩ The staff member is wearing a green vest, which is often an indicator of employee attire. They are bending towards the produce shelf. The action of bending with knees and hips flexing while maintaining a straight back is a biomechanically efficient way to lower one’s body to inspect or arrange items on lower shelves. ⟨ /think ⟩ The staff member is restocking or inspecting produce items on the lower shelf as part of routine store maintenance.
Example 2
Q: Why is this person performing the observed action?
A: ⟨ think ⟩ The customer is standing still near the dairy refrigerator with their head tilted slightly downward, scanning the shelves. This posture suggests they are searching for a specific product among many options. ⟨ /think ⟩ The person is comparing product labels to find the right variant or brand of dairy product.
Table 18
Example 1
Q: List all the people visible and their current activities in this retail scene.
A: The footage captures 2 people. Person 1: A female store employee walks along the produce display, stopping to arrange and organize vegetables on the shelves and bins. Person 2: A man walks forward in the aisle, carrying a blue plastic shopping basket.
Example 2
Q: Describe what each person is doing in this scene.
A: Three people are visible. Person 1: A customer pushes a cart through the main aisle. Person 2: A staff member arranges products on the end-cap display. Person 3: A shopper examines items on the top shelf, reaching upward.
Table 19
Example 1
Q: Describe the actions of each individual visible in this store camera view.
A: ⟨ think ⟩ From this camera view, 4 people can be identified. Person 1 walks into the produce section and interacts with staff. Person 2 is a staff member assisting at the vegetable display. Person 3 browses independently near the refrigerated section. Person 4 navigates toward the checkout area. ⟨ /think ⟩ Four individuals occupy the store with distinct goals: one customer selects produce with staff assistance, another browses independently, and a fourth heads toward checkout.
Table 20
Example 1
Q: Describe what is happening in this scene.
A: The person stands at the open refrigerator door, looking at the milk pouches and curd packages inside. The camera captures the person’s hands near the shelves. Another person walks past in the background.
Example 2
Q: Describe the environment and surroundings in this scene.
A: This is the Aisle (Sweets/Snacks) section of the supermarket. The person’s current sub-goal is: Examine Item Details and Check Shopping List.
Table 21
Example 1
Q: What is the typical arrangement of products on the shelving units in this grocery store aisle?
A: The shelving units display a variety of packaged food items, primarily snacks, arranged by category. Products are neatly stocked and often stacked to maximize shelf space, with labels facing forward for easy identification by customers.
Example 2
Q: Describe the layout of the produce section from this exocentric view.
A: The produce section features open bins along both sides of a central walkway, with fruits on the left and vegetables on the right. Overhead signage indicates product categories. A weighing scale is positioned at the end of the display.
Table 22
Example 1
Q: Are the visible items generally close to the person, or spread across a wider range?
A: ⟨ think ⟩ The depth statistics show a mean depth of 0.172. This is a relatively low value, indicating that most objects are in the closer to mid-range. The minimum depth is 0.055, suggesting something very close like the basket on the floor… ⟨ /think ⟩ Items are primarily close-range.
Example 2
Q: What is the shopper’s current state, and what does this imply for robot navigation?
A: ⟨ think ⟩ I observe a human shopper in the aisle in a “Dynamic” state. This dynamism is evidenced by the shopper actively picking up an item from a shelf. ⟨ /think ⟩ The shopper is dynamic; the robot should maintain safe distance and anticipate movement.
Table 23
Example 1
Q: Based on the spatial layout, what affordances are available to the person?
A: ⟨ think ⟩ The shelves on both sides are within immediate reaching distance, affording grasping of products like snacks, beverages, and household items. The floor affords walking through the aisle. The shopping basket provides containment for selected items. ⟨ /think ⟩ The environment affords navigation through aisles and grasping items from shelves and bins. The basket facilitates collection of products.
Table 24
Example 1
Q: Analyze the cause-and-effect: why did this action happen?
A: ⟨ think ⟩ The person is in an aisle stocked with dry goods and beverages. Their goal is to perform grocery shopping, which necessitates navigating aisles to find items on their list. Walking down this aisle directly progresses through the store’s layout toward the target items. ⟨ /think ⟩ To find items on their shopping list located in the dry goods and beverages aisle.
Table 25
Example 1
Q: Describe the typical person-object interaction required to select items from the bulk bins and refrigerated cases.
A: ⟨ think ⟩ To interact with the bulk bins, a customer would reach over the counter, lift the transparent lid, and use a scoop to retrieve the product. For refrigerated cases, the customer would open one of the glass doors to access and select items. ⟨ /think ⟩ Customers interact with bulk bins by lifting lids and scooping products, and with refrigerated cases by opening glass doors to retrieve items.
Table 26
Example 1
Q: Does this scene have a wide or narrow depth range?
A: This scene has a moderate depth range from 0.05 to 0.78. There is a mix of closer and farther objects, but the distances do not span an extreme range.
Example 2
Q: Are objects in the top portion of the view closer or farther than those at the bottom?
A: Objects in the top portion of the view are at shorter distances than those at the bottom, indicating a downward-looking perspective toward the floor and nearby shelves.
Table 27
Example 1
Q: Describe the spatial layout and obstacle proximity from this panoramic view.
A: ⟨ think ⟩ Mean depth 0.43, bimodality 0.07. The panoramic view reveals a central open aisle flanked by shelving units on both sides at moderate distance. ⟨ /think ⟩ Open central aisle, shelves on both sides at moderate distance. No immediate obstacles in the navigation corridor.
Table 28
Example 1 (Forward)
Q: Determine whether this video is playing in its original direction or in reverse.
A: This clip is playing forward. The person walks down a wide aisle, with shelves displaying biscuits and other packaged snacks on both sides.
Example 2 (Backward)
Q: Is this video playing forward or backward?
A: The video has been reversed and is playing backward. The person places the selected ghee container into the shopping trolley.
Table 29
Example 1 (Ego, Backward)
Q: Watch this video and determine if it is playing forward or backward. Analyze the body mechanics and biomechanical cues.
A: ⟨ think ⟩ While the walking motion appears smooth, the foot placement and overall gait reveal subtle inconsistencies. The natural push-off and weight transfer of a forward stride are absent, giving the movement a slightly unnatural, gliding quality. ⟨ /think ⟩ The video is playing backward.
Example 2 (Exo, Forward)
Q: Watch this video and determine if it is playing forward or backward. Analyze the temporal flow and causal relationships.
A: ⟨ think ⟩ A person enters the frame from the left and walks toward the right. The legs move in a natural forward progression. There are no visual cues of reverse motion or unnatural movement patterns. ⟨ /think ⟩ The video is playing forward.
Table 30
Example 1
Q: Something left the field of view. Does the object continue to exist when not visible?
A: The object still exists. In a retail environment, items on shelves, carts, and people maintain their physical presence even when they leave the camera frame. Object permanence is a basic physical principle.
Example 2
Q: A person walked behind a shelf and is no longer visible. Do they still exist?
A: Yes, the person continues to exist behind the shelf. Occlusion by a physical barrier does not cause objects or people to cease existing. They will likely reappear when the viewpoint changes or they move past the obstruction.
Table 31
Example 1
Q: What hand activity can you observe in this video?
A. One hand is actively engaged in gesturing while the other is still.
B. Both hands are idle and resting at the person’s sides.
C. The person is holding an object in their right hand and gesturing with the left.
D. The person is using both hands to clap rhythmically.
A: B
Table 32
Example 1 (Physics)
Q: When a shopping cart is pushed and then released on a flat store floor, it gradually slows down and stops. What is the primary reason?
A. The cart runs out of energy.
B. Friction between the wheels and the floor dissipates kinetic energy.
Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception-reasoning-decision gap in standalone Vision-Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question-answer (DQA) pipeline. Instead of passively accepting the VLM's description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.
Moving beyond the traditional paradigm of adapting internet-pretrained models to physical tasks, we present DM0, an Embodied-Native Vision-Language-Action (VLA) framework designed for Physical AI. Unlike approaches that treat physical grounding as a fine-tuning afterthought, DM0 unifies embodied manipulation and navigation by learning from heterogeneous data sources from the onset. Our methodology follows a comprehensive three-stage pipeline: Pretraining, Mid-Training, and Post-Training. First, we conduct large-scale unified pretraining on the Vision-Language Model (VLM) using diverse corpora--seamlessly integrating web text, autonomous driving scenarios, and embodied interaction logs-to jointly acquire semantic knowledge and physical priors. Subsequently, we build a flow-matching action expert atop the VLM. To reconcile high-level reasoning with low-level control, DM0 employs a hybrid training strategy: for embodied data, gradients from the action expert are not backpropagated to the VLM to preserve generalized representations, while the VLM remains trainable on non-embodied data. Furthermore, we introduce an Embodied Spatial Scaffolding strategy to construct spatial Chain-of-Thought (CoT) reasoning, effectively constraining the action solution space. Experiments on the RoboChallenge benchmark demonstrate that DM0 achieves state-of-the-art performance in both Specialist and Generalist settings on Table30.
Recent advances in large-scale pretrained vision-language-action models have improved robot policy learning, but directly deploying such policies in user-specific environments remains challenging due to limited generalization, which inevitably requires collecting a dataset tailored to the target environment. Teleoperation yields well-aligned data but is costly and difficult to scale, whereas simulation scales easily but struggles to resemble the target environment and generate task-specific trajectories. To meet both simultaneously, we propose PRISM, an end-to-end pipeline that generates personalized robotic datasets from a single image and a natural-language instruction. PRISM constructs digital cousin scenes that are semantically and geometrically aligned with the user environment yet diverse at the instance level, and synthesizes executable demonstrations without human teleoperation. Extensive experiments show that policies trained on PRISM-generated datasets outperform those trained on baseline-generated datasets on LIBERO and LIBERO-Plus, achieve up to 100% success rate on three real-world manipulation tasks, and maintain stronger performance when evaluated in environments that differ from those seen during training.