Are Foundation Models the Route to Full-Stack Transfer in Robotics?
Organizations: Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, Germany. · Stanford AI Lab, Stanford University, Palo Alto, USA.
Abstract
In humans and robots alike, transfer learning occurs at different levels of abstraction, from high-level linguistic transfer to low-level transfer of motor skills. In this article, we provide an overview of the impact that foundation models and transformer networks have had on these different levels, bringing robots closer than ever to "full-stack transfer". Considering LLMs, VLMs and VLAs from a robotic transfer learning perspective allows us to highlight recurring concepts for transfer, beyond specific implementations. We also consider the challenges of data collection and transfer benchmarks for robotics in the age of foundation models. Are foundation models the route to full-stack transfer in robotics? Our expectation is that they will certainly stay on this route as a key technology.
Figures & tables
| Transfer Learning “leverages prior knowledge from a source space, composed by a robot, a task, and an environment, to improve the performance in a target space, where one or more modes differs from the source space.” [ 1 ] |
| Foundation models are “trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.” [ 8 ] . This definition applies to all LLMs, VLMs and VLAs cited in this paper. |
| A Vision Language Action model (VLA) “is a system that takes visual observations and natural language instructions as required inputs and may incorporate additional sensory modalities. It produces robot actions by directly generating control commands.” [ 2 ] . In Section 4 , we adhere to the classification of VLA architectures in Fig. 4 in [ 2 ] . |
Transfer Learning
Transfer learning studies how models reuse knowledge across tasks, domains, or modalities when target data or computation are limited.
See all 417 papers