Jeff Dean: The 1% Rule for Building in AI
Y Combinator Startup PodcastFull Title
Jeff Dean: The 1% Rule for Building in AI
Summary
This episode features Jeff Dean discussing the evolution of AI, from his past contributions to current trends and future predictions. He emphasizes the importance of systems thinking, efficient hardware, and the potential for AI to self-improve, while also offering advice for aspiring founders on identifying impactful problems and developing "taste" in their work.
Key Points
- AI models are rapidly approaching and, in some cases, exceeding the capabilities of a junior engineer, with significant advancements in agent-based and longer-running coding tasks.
- The future of AI development will likely involve significant automation of ML systems themselves, enabling them to improve their own capabilities through experimental loops and problem decomposition.
- The "fits in memory" moment for AI development today is focused on high-performance, low-energy inference hardware, crucial for making agent-based systems accessible and responsive.
- Long-running AI agent tasks, capable of spanning days or weeks, are becoming feasible for complex software implementation and other sophisticated problems, challenging the assumption that agents are limited to short durations.
- Specialized hardware, like TPUs, designed for efficient linear algebra operations, have proven foundational for machine learning, highlighting the benefits of hardware specialization for AI workloads.
- The core of AI development is increasingly viewed as a compression problem, where understanding data allows for efficient representation and subsequent restoration, with transformer architectures being a key enabler of this.
- AI progress is shifting from solely focusing on model size and parameters to a holistic approach encompassing retrieval, tools, memory, and agent orchestration, requiring complex systems engineering.
- Context engineering, which involves providing models with relevant information and guidelines to enhance their capabilities, is an accessible area for founders to innovate without needing massive computational resources.
- For startup founders, identifying niche domains where general AI models currently struggle or lack access to specific data is a promising avenue for competitive advantage, though the rapid improvement of general models requires careful consideration of long-term viability.
- The scarce skill for founders in the age of AI agents is not coding, but rather developing "taste"—the wisdom to identify impactful problems and guide AI-assisted computation effectively.
- Developing "taste" involves gaining experience, reflecting on industry trends, and engaging in "crazy thought experiments" that question fundamental assumptions, similar to how MapReduce and TPUs originated.
- The scientific method, when automated and accelerated by AI for running numerous experiments, has the potential to revolutionize fields like machine learning, science, and engineering by dramatically speeding up discovery and design.
- Distillation, a technique for training smaller, more efficient models from larger ones, despite initial rejection from academic conferences, has become a critical tool for deploying capable AI widely, underscoring the importance of persistence.
- The most impactful problems to work on are those that, if solved, would significantly improve the world, and it's crucial to work with enjoyable colleagues and hard problems that foster personal and collective growth.
Conclusion
The rapid advancement of AI necessitates a shift in focus from just model capabilities to sophisticated systems engineering, including hardware, data management, and agent orchestration.
Aspiring founders should prioritize identifying genuinely impactful problems that general AI models cannot yet solve, and developing "taste" in problem selection is a critical scarce skill.
Persistence in research and development, even in the face of setbacks or rejection, is crucial for breakthroughs that can significantly shape the future of technology and its impact on the world.
Discussion Topics
- What are the most significant limitations of current AI agents, and what breakthroughs are needed to overcome them for truly long-running, complex tasks?
- How can aspiring AI founders best identify "taste" in problem selection, and what are the most promising niche domains for specialized AI development in the next 3-5 years?
- Given the increasing importance of hardware and systems engineering in AI, what are the key "latency numbers" and efficiency metrics that engineers and founders should prioritize today?
Key Terms
- MapReduce
- A programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.
- Bigtable
- A distributed storage system for managing structured data that is designed to scale massively.
- TensorFlow
- An end-to-end open-source platform for machine learning, enabling researchers and developers to build and deploy ML-powered applications.
- TPU (Tensor Processing Unit)
- Google's custom-designed application-specific integrated circuit (ASIC) used for machine learning and deep neural network computations.
- Gemini
- A family of large multimodal models developed by Google AI, capable of understanding and operating across different types of information including text, code, audio, image, and video.
- Inference
- The process of using a trained machine learning model to make predictions on new, unseen data.
- Latency
- The time delay between the cause and the effect of some physical change in the system being observed. In computing, it refers to the time it takes for a data packet to get from one point to another.
- Agent-based systems
- Computer systems that can perceive their environment and take actions to achieve goals, often interacting with other agents or systems.
- Linear Algebra
- A branch of mathematics concerning linear equations such as y = mx + b, systems of linear equations, and their representations in terms of transformations of vector spaces.
- High-precision
- In computing, refers to numbers stored with a large number of digits, offering greater accuracy.
- Low-precision
- In computing, refers to numbers stored with fewer digits, offering less accuracy but faster computation and lower memory usage.
- GPUs (Graphics Processing Units)
- Computer hardware designed to accelerate graphics rendering, but also widely used for parallel computing tasks like those in machine learning.
- Transformer Architecture
- A neural network architecture introduced in 2017 that relies heavily on a mechanism called "self-attention" to weigh the importance of different words in a sequence, revolutionizing natural language processing.
- ML (Machine Learning)
- A type of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so.
- Epoch
- In machine learning, one epoch refers to the completion of one full cycle through the training dataset.
- Batching
- A technique in machine learning where multiple data samples are processed together as a single batch, rather than one by one.
- Data IO (Input/Output)
- The process of transferring data between a computer and the outside world, such as from storage devices to main memory.
- Context Engineering
- The practice of providing relevant context to large language models to improve their performance and guide their responses.
- Distillation
- A machine learning technique where a smaller, more efficient "student" model is trained to mimic the behavior of a larger, more complex "teacher" model.
- Neuromorphic Computing
- A type of computing that mimics the structure and function of the human brain.
- MapReduce
- (Already defined above)
- Functional Languages
- Programming languages that treat computation as the evaluation of mathematical functions and avoid changing state and mutable data.
Timeline
Jeff Dean discusses the prediction that AI is at the level of a junior engineer and how current models are exceeding expectations in coding and other agent-based tasks.
Jeff Dean predicts that by 2027, ML systems will be significantly automated, using AI to improve their own capabilities through experimentation and problem decomposition.
The discussion revisits the historical "fits in memory" moment for Google Search, highlighting a past innovation that dramatically improved performance.
Jeff Dean identifies high-performance and low-energy inference hardware as the current "fits in memory" moment in AI development, emphasizing its importance for agent-based systems.
Jeff Dean points out that a common false assumption about AI is underestimating the possibility of long-running, complex agent-based systems.
Examples of complex tasks that AI agents can perform include implementing entirely new software versions in different programming languages with improved safety and performance.
The origin story of the TPU is recounted, stemming from a back-of-the-envelope calculation about the computational cost of widespread speech recognition use.
TPUs were designed as specialized hardware for low-precision dense linear algebra, proving highly effective for machine learning inference.
For aspiring founders, the advice is to identify bottlenecks and consider radically different solutions that offer orders-of-magnitude improvements, rather than simply optimizing current approaches.
The AI edition of "Latency Numbers Every Engineer Should Know" focuses on metrics like memory bandwidth, energy per operation, and interconnect bandwidth between chips.
Energy is highlighted as a primary metric for AI system design, with data movement (IO) being orders of magnitude more energy-intensive than computation itself.
Batching, commonly used in training models, is presented as a system/data-IO problem to improve hardware efficiency, not solely a model problem.
Jeff Dean is focusing on inference rather than training for new hardware specializations, emphasizing the need for very low latency.
AI is framed as a large compression problem, where understanding data is key to effective compression and restoration.
AI progress is now seen as encompassing more than just model improvements; it includes retrieval, tools, memory, and agent orchestration, requiring complex engineering.
Becoming exceptional at context engineering involves using AI models to solve problems, observing failures, and iteratively improving prompt design and tool integration.
An example of context engineering is creating a skill that automates performance optimization by teaching an AI to perform microbenchmarking and iterative code modifications.
Agents can falter due to errors compounding in an open-loop system or when they encounter situations outside their training distribution.
Strategies to improve agent reliability include providing skills and hints to keep them on a "brightly lit path" and using multi-agent systems with evaluators to search for optimal solutions.
Startups can win by focusing on specific domains where general models lack depth or data, creating specialized models or delightful user experiences.
Founders should look for problems where general models currently succeed only 0-1% of the time, indicating a gap that specialized solutions can fill.
Niche domains like protein folding (AlphaFold), material science, and chip design can benefit from highly accurate, specialized models.
Managing fleets of AI agents effectively requires clear, crisp design documents and specifications to guide their actions.
The scarce skill for founders is not in managing agents or writing code, but in having "taste"—the high-level wisdom to choose impactful problems.
Developing "taste" comes from experience, reflecting on past predictions and industry trends, and engaging in "crazy thought experiments."
A "crazy thought experiment" involves questioning fundamental assumptions, such as the assumption of perfect reliability in transistors, to explore new design paradigms.
Past consequential systems like TPUs and MapReduce originated from thought experiments that challenged existing assumptions.
The foundation of the scientific method, when automated and accelerated by AI, is poised to tackle complex problems in ML, science, and engineering through rapid experimentation.
Faster validation models, trained approximations of computationally intensive simulators, can drastically accelerate scientific discovery loops.
Key areas for AI-driven self-improvement include accelerating machine learning model design, discovering new data-efficient algorithms, and advancing multi-agent interactions and human discourse.
The paper on distillation, initially rejected, became a foundational technique for creating efficient AI models, highlighting the importance of persistence despite setbacks.
If a young Jeff Dean were transported to today, he would focus on working on problems he cares about that can make a positive difference in the world, collaborating with enjoyable colleagues.
Lessons for working with smart people include finding individuals with complementary skills, low ego, team-player attitudes, and whom one enjoys collaborating with.
Jeff Dean hopes future consequential systems will address challenges in new hardware approaches, more data-efficient ML algorithms, continuous learning, multi-agent interactions, and improving human discourse.
Episode Details
- Podcast
- Y Combinator Startup Podcast
- Episode
- Jeff Dean: The 1% Rule for Building in AI
- Official Link
- https://www.ycombinator.com/
- Published
- August 1, 2026