Robot-Use Agents: Why General-Purpose Models May Win in Robotics...
Y Combinator Startup PodcastFull Title
Robot-Use Agents: Why General-Purpose Models May Win in Robotics
Summary
This episode explores the emerging era of "robot-use agents" where general-purpose large language models (LLMs) are enhancing robot capabilities.
Founders from Waterlabs and Robocurve discuss how LLMs are being integrated into robotics, the evolution of VLA models, and the potential for general-purpose robots to perform tasks akin to a competent teenager within the next two years.
Key Points
- General-purpose language models are proving effective at controlling robots, leading to the concept of "robot-use agents" that can make various robots more capable.
- Early research, like the RT2 paper, demonstrated fine-tuning pre-trained language models to output robot control commands (pose), laying the groundwork for current advancements.
- The development of VLAs (Vision-Language Action models) and the evolution of LLMs to directly output code or engage in "chain of thought" reasoning have significantly improved their ability to control robots compared to earlier, more constrained models.
- The concept of "code as policies," where LLMs generate code to control robots, is a key development, allowing for more complex and on-the-fly tool creation and utilization, exemplified by projects like Voyager.
- The effectiveness of LLMs in robotics is not solely dependent on architectural changes but crucially on the quality and type of training data, with a shift towards leveraging vast web data and computer vision data for improved spatial understanding.
- In-context learning (ICL) allows LLMs to adapt quickly to new tasks by providing examples in the prompt, but it has limitations in performance and context window size, suggesting that direct weight updates or more robust data integration methods are still necessary for complex, long-term learning.
- Frameworks like the "harness" developed by Waterlabs aim to consolidate learned skills and experiences into reusable programs, enabling faster and more efficient execution of tasks by robots.
- The "Platonic Representation Hypothesis" suggests that LLMs trained on diverse data converge to consistent world representations, implying that powerful general-purpose LLMs inherently possess strong robotics capabilities.
- Advanced models like Astra show significant improvements in spatial intelligence due to extensive pre-training on computer vision and CAD data, which imbues them with a better understanding of physical environments and manipulation.
- The rapid improvement in LLM latency is driving the feasibility of real-time robot control, with predictions of real-time capabilities by the end of the year if current trends continue.
- The development of general-purpose robots capable of performing tasks comparable to a competent teenager is anticipated within the next two years, a development that society may not be fully prepared for.
Conclusion
General-purpose LLMs are rapidly advancing robotics, leading to the imminent arrival of "robot-use agents."
The integration of diverse data sources, particularly computer vision and coding data, is crucial for developing robots with sophisticated spatial understanding and manipulation abilities.
Within the next two years, we can expect the emergence of general-purpose robots capable of performing a wide range of tasks, a significant development that society may not yet fully grasp.
Discussion Topics
- How can the current advancements in LLMs and robotics be effectively communicated to the general public to prepare them for the impending arrival of general-purpose robots?
- What are the ethical considerations and societal impacts of having robots capable of performing tasks at the level of a competent human teenager?
- Beyond task execution, what are the most exciting or unexpected applications of these advanced robot-use agents that researchers are currently exploring?
Key Terms
- LLM
- Large Language Model. A type of artificial intelligence trained on vast amounts of text data to understand and generate human-like language.
- VLA
- Vision-Language Action model. A model that can process visual input, understand language instructions, and generate actions for a robot.
- Pose
- In robotics, the position and orientation of an object or robot end-effector in space.
- RT2
- A specific model architecture developed by Google that integrates vision, language, and action capabilities for robot control.
- Chain of Thought (CoT)
- A prompting technique that encourages language models to break down complex problems into intermediate steps, improving reasoning and performance.
- In-context learning (ICL)
- A method where a language model learns to perform a task by being provided with a few examples in its input prompt, without updating its internal weights.
- Harness
- In this context, a software framework or system that facilitates the interaction between an LLM and a robot, often by providing standardized interfaces and tools.
- Platonic Representation Hypothesis
- A theoretical idea suggesting that large neural networks trained on diverse data develop internal representations of concepts that are consistent across different modalities and tasks.
- Transduction
- A machine learning paradigm where a model learns a direct mapping from input to output, often requiring large amounts of data.
- Program Induction
- The process of inferring a program that performs a specific task, often by learning from examples.
- CAD data
- Computer-Aided Design data, typically used for creating 3D models of objects and environments.
- GUI
- Graphical User Interface. A type of user interface that allows users to interact with electronic devices through graphical icons and visual indicators.
- Dagger
- A data segregation framework often used in Reinforcement Learning (RL) that involves collecting data, reflecting on it, and using it to update model weights.
Timeline
MIT professor Philip Isola suggests the era of "robot-use agents" is emerging.
Founders of Waterlabs and Robocurve introduce their companies and focus on LLMs for robot control and physical AI benchmarking.
Viral videos showcasing LLMs controlling robots for tasks like unscrewing caps and inter-robot communication are discussed.
The discussion delves into early research on pre-training language models for decision-making and the RT2 paper, which established VLAs.
Jay explains the RT2 paper's approach of using a fine-tuned language model to output robot poses instead of English.
The concept of "chain of thought" reasoning in LLMs is explained using a math problem example and its parallel to robotics tasks.
The importance of giving LLM agents more autonomy and resources for improved capabilities is highlighted.
Hameng explains that RT2's effectiveness comes from leveraging the vast data of pre-trained language models.
Jay elaborates on improvements in tool use and complex policy generation as code as key differences since RT2.
The concept of "code as policies" and its inspiration for current robotics research is introduced.
Jay discusses Voyager and the idea of coding agents requiring good tool use and on-the-fly tool creation.
Early "code as policies" papers, particularly from Google's D1 team, are described as creating functions for coding agents to control robots.
The one-shot and in-context exploration abilities of coding agents in these early papers are highlighted as a motivation for further LLM research in robotics.
Francois introduces his framework of learning embedded in weights versus in-context learning and how past research fits into it.
Francois discusses an experiment measuring the efficiency of inputting learning data into a policy, comparing in-context learning to other methods.
Hameng and Jay discuss the concept of a "harness" as a form of domain specificity and consolidation of learned skills.
A demonstration of Astra controlling robot arms to move a block is shown and discussed.
The difference between directly using a model like Astra and using a harness API like Waterlabs' is explained.
The steps involved in a coding agent controlling a robot are walked through, contrasting direct coding agent use with a harness loop.
Hameng explains the process of Astra receiving images and outputting end-effector poses for robot control.
The current process is described as more full calls than "code as policies," with opportunities for variation and more flexible responses.
The biggest insight for achieving AGI is discussed, relating to transduction versus program induction and the role of code as a generator.
The "Platonic Representation Hypothesis" is introduced, suggesting that LLMs converge to consistent world mappings regardless of training data.
The speakers discuss why Astra is particularly good at spatial intelligence and what might have changed in OpenAI's approach.
Astra's improved vision capabilities, potentially due to more computer vision and CAD data in its pre-training, are highlighted.
The utility of computer vision data, like navigating GUIs, for understanding spatial concepts relevant to robotics is explained.
The idea of feeding all kinds of data, including coding and computer use data, into a single model to achieve the most capable robot use is proposed.
The discussion shifts to future capabilities, with the expectation of general-purpose robots within two years.
"General purpose robots" are defined as being able to perform any task a competent teenager could with their hands, generalizing to unseen situations.
Challenges like latency in robot control are discussed, and the need for consolidating initial passes into faster, scalable policies.
The concept of compression and distillation from daily experiences to train weights, similar to human learning, is proposed for robotics.
The discussion touches on Dreamcoder-esque approaches and refactoring traces or skills for more efficient robotic policy learning.
Future challenges in managing growing content of skills and deployment data are mentioned.
Episode Details
- Podcast
- Y Combinator Startup Podcast
- Episode
- Robot-Use Agents: Why General-Purpose Models May Win in Robotics
- Official Link
- https://www.ycombinator.com/
- Published
- September 26, 2026