Foundation Model Driven Robotics: A Comprehensive Review
Organizations: School of Mechanical and Aerospace Engineering, Nanyang Technological University, 639798, Singapore. · J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77801, USA.
Abstract
The rapid emergence of foundation models, particularly large language models and vision-language models, has introduced a transformative paradigm in robotics. The models offer powerful capabilities in semantic understanding, high-level reasoning, and cross-modal generalization, enabling significant advances in perception, planning, control, and human-robot interaction. This critical review provides a structured synthesis of recent developments and organizes applications into four directions: simulation-driven robotics, open-world operation, sim-to-real transfer, and adaptable robotic systems. Unlike existing surveys that emphasize isolated capabilities, this work highlights integrated, system-level strategies and evaluates their practical feasibility in real-world environments. Key enabling trends such as procedural scene generation, policy generalization, and multimodal reasoning are discussed alongside core bottlenecks, including limited embodiment, lack of multimodal data, safety risks, and computational constraints. Through this lens, this paper identifies both the architectural strengths and critical limitations of foundation model-based robotics, highlighting open challenges in real-time operation, grounding, resilience, and trust. The review concludes with a roadmap of proposed future research directions for bridging semantic reasoning and physical intelligence through more robust, interpretable, and embodied models.
Figures & tables
| Model | Architecture | Key Capabilities | Applications in Robotics | Reference |
| Large Language Models (LLMs) | ||||
| BERT (2018) | Transformer encoder (bidirectional); 110M/340M parameters | Contextual word representations; strong language understanding for natural language processing (NLP) tasks | Parses robotic task steps into structured representations (e.g., behavior trees) for adaptive planning; supports question answering (QA) and dialogue in service robots. | [ 43 , 44 , 45 ] |
| T5 (2019) | Encoder-decoder; up to 11B parameters | Unified text-to-text framework; strong transfer learning capabilities | Enables flexible symbolic task translation, skill sequencing, motion generation, and cross-modal grounding in language-to-action and action-to-language tasks. | [ 46 , 47 , 48 , 49 ] |
| Codex (2021) | Decoder (GPT-3-based) | Code generation from natural language instructions | Translates natural language into robot control code for manipulation, path planning, and feedback systems in simulators and real-world tasks. | [ 50 , 13 , 51 , 52 ] |
| GPT-3.5 (2022) | Decoder (Transformer; GPT-3 lineage); size undisclosed | Instruction-tuned model with high adaptability and reasoning capabilities | Enables real-time path planning, tool retrieval correction, and multi-robot control via natural language commands in simulation and physical setups. | [ 53 , 54 , 55 ] |
| InstructGPT (2022) | Decoder (GPT-3 fine-tuned with reinforcement learning from human feedback) | Aligned with human preferences; refined instruction following | Interactive planning and code generation for manipulation and navigation; context-aware object embeddings; perception-planning-action loops via Python (e.g., text-davinci-003 + Segment Anything Model (SAM)/CLIP) | [ 56 , 57 , 58 ] |
| Subsystem | Humanoids | Manipulators | Wheeled Robots | Quadrupeds |
|---|---|---|---|---|
| Perception | Scene-aware object disambiguation from dialogue (e.g., “pick up the cup to the left of the book”) | Multi-view affordance detection using open-vocabulary vision models | Semantic navigation via open-set object recognition | Terrain-aware object detection for unstructured or outdoor environments |
| Planning | Task decomposition for high-level instructions (e.g., “set the table for dinner”) | Assembly planning using LLM-generated action sequences and tool affordances | Navigation plan synthesis from natural language goals (e.g., “go to the kitchen”) | Path generation incorporating terrain constraints and verbal mission goals |
| Control | Adaptive motion generation from high-level prompts (e.g., “walk over and wave”) | Low-level control code synthesis for stacking, sorting, or manipulating parts | Dynamic obstacle avoidance based on text-guided policy modulation | Gait and balance adaptation via LLM-suggested behavior strategies |
| Interaction | Multimodal dialogue including gesture, gaze, and context-rich conversation | Instruction refinement through natural language correction (e.g., “place it upright”) | Conversational feedback during navigation (e.g., “turn after the red sign”) | Remote mission updates via verbal feedback or contextual corrections |
Robot Foundation Models
Robot Foundation Models are pretrained systems that combine perception, reasoning, and action to support diverse tasks and robot embodiments.
See all 151 papers