20VC: Who Wins the Inference Market | Has Open-Source LLM Inference...

Full Title

20VC: Who Wins the Inference Market | Has Open-Source LLM Inference Peaked? | What Happens to the Cost of Compute | The Real Limits of Post-Training and RL with Erik Bernhardsson, Modal CEO

Summary

Erik Bernhardsson, CEO of Modal, discusses the rapidly evolving inference market, highlighting the immense opportunities and challenges in compute, model specialization, and platform development.

The conversation also touches on founder journeys, the importance of brand, and the future of AI-driven applications and infrastructure.

Key Points

• Early passion for coding and building, stemming from childhood, laid the foundation for a successful entrepreneurial career, with a realization that enjoying the work is paramount.
• The ideal age to start a company is considered to be around 32, balancing youthful ambition with acquired experience in navigating business complexities.
• The prevalent "eating glass" mentality in startup culture is dismissed by the founder, who prioritizes daily enjoyment and building over suffering for a distant reward.
• Fundraising is identified as a less enjoyable aspect of entrepreneurship, though the process is streamlined through building strong relationships well in advance of capital needs.
• Price optimization in fundraising is seen less as a race for the highest valuation and more about strategic capital raising to maintain investor hunger and avoid excessive dilution.
• The company has maintained capital efficiency, with a recent increase in spend on GPU capacity becoming a significant factor in cash flow.
• The concept of a "moat" in early-stage companies is downplayed, with the emphasis shifting to speed, a strong team, and brand as key determinants of defensibility.
• Building a strong brand and community around a company is crucial for differentiation, requiring authenticity and a distinct personality that resonates with users.
• The inference market is predicted to become enormous, akin to the new cloud, but success will require a comprehensive platform of integrated services beyond just inference.
• The commoditization of single-purpose AI products is a concern, emphasizing the need for a suite of tools to build robust AI applications and create switching costs.
• The current software margin landscape has shifted significantly, with the traditional high-margin models largely gone due to the layered vendor stack.
• The GPU market is described as chaotic and competitive, with Modal managing capacity across multiple cloud providers to serve its customers efficiently.
• Modal's competitive advantage lies in its proprietary infrastructure, allowing it to treat global compute as a unified pool and eliminate capacity planning burdens for its clients.
• The future of the inference market will involve increased specialization, with opportunities for both large-scale infrastructure providers and niche players.
• The CUDA ecosystem is recognized as a significant, albeit sometimes frustrating, advantage for NVIDIA, driving broader support and optimization for their hardware.
• Neo-clouds are seen as both friends and foes, with their attempts to move up the stack being challenging but also potentially creating opportunities for specialized infrastructure providers.
• Open-source models will continue to thrive, especially for companies seeking fine-tuning and greater control, while proprietary models will also find a strong market.
• The trend towards specialization in the chip market, with players like Cerebrus and Etch, coexists with NVIDIA's continued dominance, suggesting a market for diverse hardware solutions.
• The difficulty in replicating Modal's complex infrastructure stack, built over five years, provides a strong engineering-based moat.
• The financial engineering involved in scaling neo-clouds, securing contracts, and managing capital is a critical, often overlooked, aspect of their success.
• The long-term outlook for software engineers remains very bullish, as increased efficiency leads to new applications and sustained demand.
• The role of software engineers is evolving beyond simple code translation to understanding requirements and acting as interfaces between humans and complex systems.
• The increasing use of AI agents in coding is seen as a significant development, but not a replacement for human engineers, who operate on longer time scales and with complex feedback loops.
• The trend towards post-training AI models is driven by the ability to create finite feedback mechanisms, particularly in areas like coding and cybersecurity.
• The inference market is experiencing significant fragmentation, leading to opportunities for specialized players and a need for comprehensive platforms that offer a breadth of services.
• The current GPU market is characterized by high demand and prices, but an expectation of eventual depreciation as supply catches up.
• The growth of the AI market is expected to continue exponentially, with potential pullbacks in the stock market being temporary given the vast long-term opportunity.
• Data marketplaces and services are evolving to offer full-lifecycle solutions, from acquisition to deployment and enterprise-level support.
• The concept of "owning your own intelligence" is beneficial for some use cases, but not universally applicable, with factors like latency, cost, and platform integration also being critical.
• Model routing, while a potentially lucrative standalone business, is likely to become a feature within more comprehensive AI platforms in the long term.
• The increasing cost of compute, particularly GPUs, will drive innovation and competition in the inference market.
• The talent market for exceptional engineers is extremely competitive, with high compensation packages offered by leading AI labs.
• Modal aims to provide a comprehensive platform of products and services to meet diverse developer needs, fostering a broad ecosystem.
• The shift in resource allocation from pre-training to post-training AI models reflects the growing demand for practical AI applications with clear feedback mechanisms.
• The magnitude of the AI market growth is underestimated, with both open-source and frontier model providers expected to see substantial expansion.
• The increasing efficiency of AI tools will likely lead to a surge in demand for software and continued relevance for software engineers.
• The success of Modal's sandbox product demonstrates the importance of building robust infrastructure components that can scale with emerging AI trends.
• The long-term trend of hardware prices decreasing is expected to continue, despite current supply constraints.
• The future of drug discovery is seen as a prime area for AI application, despite regulatory hurdles.
• The competitive landscape in AI is dynamic, with both large players and specialized startups carving out niches.

Conclusion

The inference market is experiencing explosive growth and will continue to be a massive opportunity, requiring comprehensive platform solutions rather than single-point products.

Building a strong brand and community, alongside a relentless focus on customer experience and product development, are critical for long-term success in the AI space.

The future of AI will create unprecedented demand for compute and sophisticated applications, ensuring continued relevance for infrastructure providers and software engineers.

Discussion Topics

• What are the biggest challenges and opportunities in the rapidly evolving AI inference market over the next five years?
• How does a strong company brand and community impact defensibility and long-term growth in the AI sector?
• What is the future of software engineering in an era of increasing AI automation and agent-assisted development?

Key Terms

Inference

The process of using a trained machine learning model to make predictions or decisions on new, unseen data.

ARR

Annual Recurring Revenue, a metric used to track the predictable revenue a company expects to receive from its customers over a year.

VC

Venture Capital, funding provided by investors to startups and small businesses with perceived long-term growth potential.

PLG

Product-Led Growth, a go-to-market strategy where customers can discover, try, and buy the product themselves, often starting with a free trial.

GPU

Graphics Processing Unit, a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. Increasingly used for AI model training and inference.

CVE

Common Vulnerabilities and Exposures, a list of publicly disclosed security vulnerabilities.

LLM

Large Language Model, a type of artificial intelligence model trained on massive amounts of text data to understand and generate human-like language.

SOC2

Service Organization Control 2, a compliance framework for service organizations that assures customers that their data is secured and managed properly.

ISO 27001

An international standard for information security management systems.

HIPAA

Health Insurance Portability and Accountability Act, a US law that sets standards for the protection of sensitive patient health information.

GDPR

General Data Protection Regulation, a comprehensive data privacy law in the European Union.

Open Source

Software with source code that anyone can inspect, modify, and enhance.

Frontier Models

State-of-the-art AI models, often proprietary and developed by leading research labs, that push the boundaries of AI capabilities.

Reinforcement Learning (RL)

A type of machine learning where an agent learns to make a sequence of decisions by trying to maximize a reward it receives for its actions.

Sandboxes

Isolated environments where code can be executed safely without affecting the host system, often used for testing and development.

Jevons Paradox

The principle that increasing the efficiency with which a resource is used tends to increase, rather than decrease, the rate of consumption of that resource.

Moat

A sustainable competitive advantage that protects a company from competitors.

IPO

Initial Public Offering, the first time a company offers shares of stock to the public.

Secondaries

The sale of existing shares of a company's stock by current shareholders, rather than newly issued shares.

Timeline