AI Can Write the Code. Can It Run the System?

Full Title

AI Can Write the Code. Can It Run the System? | Grepr CEO Jad Naous

Summary

This episode explores the evolving capabilities of AI, particularly its potential to not only write code but also to manage and run complex operational systems.

The discussion highlights the current limitations and future possibilities of AI in areas like infrastructure management, cybersecurity, and software development, emphasizing the distinction between AI's capabilities in simulated digital environments versus the complexities of the real world.

Key Points

AI's advancement in code generation has surpassed initial expectations, moving from simple autocompletion to modeling greater system complexity, demonstrating AI's ability to break through perceived limitations.
Running complex distributed systems has historically relied heavily on human intervention, indicating a significant gap between AI's current coding abilities and its capacity to manage live, unpredictable operational environments.
The core challenge in applying AI to operations lies in its ability to reason about real-world complexity and make decisions when the answer isn't explicitly present in its training data, contrasting with its success in well-defined coding tasks.
The guest's company, Grepr, aims to reduce observability costs by extracting signals upstream and feeding them into AI systems, thereby improving reliability and operations.
While AI excels at syntax generation (writing code), its ability to grasp semantics (how the system behaves) and make nuanced operational trade-offs is still developing, especially in complex distributed systems.
The adoption of AI tools by engineers varies significantly, with highly skilled engineers leveraging them to become even more effective, while less adaptive engineers may see less benefit.
AI has dramatically increased individual engineering productivity, with complex projects that once took months for a team now being completed by one person in days.
Applying AI to operational systems is proving more challenging than to code generation because operational environments are often heavy-tailed, exception-based, and less predictable than code itself, making them difficult for AI to model and manage.
While AI can assist in identifying anomalies and potential issues in operations and security, the human ability to reason about novel or complex situations and to build trust remains critical for managing real-world systems.
The economic viability of AI solutions, particularly for real-world problems like self-driving cars, is a significant factor determining their widespread adoption, suggesting that AI's impact will be most profound where it creates clear economic advantages.

Conclusion

AI has made significant strides in code generation, exceeding initial expectations and increasing developer productivity, but applying it to complex, unpredictable operational systems remains a considerable challenge.

The distinction between AI operating in a digital, simulated environment versus the messy, heavy-tailed nature of the real world is critical for understanding its current limitations in running and managing systems.

While AI is a powerful tool for analysis and identifying patterns, human judgment, trust-building, and the ability to navigate unforeseen circumstances are still indispensable for effective system operations and strategic decision-making.

Discussion Topics

How can AI bridge the gap between generating code and effectively managing the complex, real-world operations of software systems?
What are the most promising near-term applications of AI in improving system reliability and operational efficiency, beyond code generation?
Given the challenges of applying AI to the unpredictable nature of real-world systems, what role will human expertise and judgment continue to play in the future of IT operations?

Key Terms

Observability: The ability to understand the internal state of a system by examining its outputs, crucial for debugging and performance analysis.
Telemetry: Data collected from systems, such as logs, metrics, and traces, used for monitoring and analysis.
Microservices: An architectural style that structures an application as a collection of small, independent services.
Semantics: In computer science, the meaning of code or commands, defining what the machine actually does.
Syntax: In computer science, the arrangement of symbols and words in a programming language according to a set of rules.
WSL (Windows Subsystem for Linux): A compatibility layer for running Linux binary directly on Windows.
Data Lake: A centralized repository that allows you to store all your structured and unstructured data at any scale.
Parquet: A columnar storage file format that offers optimizations to speed up analytics.
Iceberg: An open table format for huge analytic datasets.
Auto-promotion (in data lakes): The process of extracting and optimizing data fields from semi-structured logs to become independent columns for efficient querying.
Chaos Engineering: A discipline of experimenting on a system in order to build confidence in its capability to withstand turbulent conditions in production.
TAM (Total Addressable Market): The total market demand for a product or service.
Universal Function Approximator: A concept in machine learning that a neural network with a given set of activation functions can approximate any continuous function.

Timeline

00:08:36

AI's ability to model complexity in systems has broken through initial expectations.

(00:44:840) The "dirty secret" of running distributed systems has long been human management.

(01:54:520) AI's capability is tested when answers are not contained within its training data.

(07:36:496) Engineering leaders are seeking solutions for expensive infrastructure and observability costs, aiming to buy reliability.

(08:16:816) The shift to microservices has increased data volume, making traditional observability less cost-effective for reliability.

(09:35:976) Grepr aims to reduce telemetry costs and feed signals into AI for reliability operations.

(10:37:696) The discussion shifts to how AI is changing software development and engineering practices.

(11:22:576) AI in coding has evolved from autocompletion to more sophisticated assistance, with good engineers adapting it into their workflows.

(12:01:536) The effectiveness of AI interaction depends on understanding specific models and harnesses.

(13:28:712) A disparity in AI adoption and effectiveness is emerging between engineers.

(14:00:672) Understanding complex system setups is crucial for leveraging AI effectively.

(14:50:552) Semantics in computer science refer to what the machine actually does, distinct from syntax.

(18:10:272) The conversation focuses on the practical application of AI in project development and its time-saving benefits.

(18:39:152) The guest details his current tech stack: Windows, WSL, Cloud Code, and Opus 5.5.

(19:47:248) The use of Windows with WSL and Cloud Code is highlighted for development.

(20:14:848) The guest uses Opus 5.5 and has shipped significant code updates rapidly.

(21:08:968) Advanced techniques for using AI models like Claude involve managing persistent sessions and sub-agents.

(21:35:528) The guest typically manages three to four concurrent AI session windows to maintain context.

(22:00:728) AI agents interact directly through the command line, with one agent acting as a tech lead reviewer.

(23:20:488) A recent project on auto-promotion in data lakes was completed by one person in 10 days, a task that previously took six months.

(25:05:168) A key question is how AI can reason about operational systems and make necessary trade-offs, given the complexity and unpredictability of the real world.

(26:54:927) The challenge is converting operational complexities into something an AI can reason about.

(27:34:247) The current limitations of AI in predicting system behavior and operational correctness are discussed.

(29:34:848) AI can optimize code for specific benchmarks and queries when grounded in performance data.

(30:02:288) AI is performing well in identifying operational changes but struggles with predicting resource allocation like memory.

(30:35:326) AI is being used for operational and security tasks, but its application differs from traditional lagging-indicator approaches.

(31:58:126) AI models are proving effective in identifying system anomalies and potential issues by detecting changes in log patterns.

(32:51:846) The distribution of training data for code generation is manageable, unlike the complex, heavy-tailed distribution of operational environments.

(33:48:126) The application of AI to run distributed clusters is still an open question, with human intuition and experience playing a crucial role.

(35:36:685) The goal for AI in operations is to identify changes and follow a workflow to diagnose issues, rather than just predicting outcomes.

(37:00:644) AI is expected to significantly improve query optimization in the near future.

(37:24:406) The history of AI in coding, from autocomplete to complex system building, shows a progression beyond initial expectations.

(38:18:644) Operational environments are seen as more complex and less predictable for AI than code generation due to their heavy-tailed and exception-based nature.

(39:02:644) The historical trend in systems is to over-provision rather than relying on complex predictive algorithms, and operations remain a more challenging domain for AI.

(40:05:004) The focus for AI in operations is on detecting changes and understanding the underlying causes, not just predicting future states.

(41:06:524) AI's role is to follow established debugging workflows to pinpoint issues, making complex problems manageable.

(42:02:283) Applying AI to security involves identifying behavioral anomalies and tracing potential causal chains, which is complex due to the adversary's role.

(43:07:524) The inherent complexity and vast state space of operational systems make them harder for AI to master than code.

(44:46:921) The finiteness of digital data, like MP3s, offers a manageable distribution for AI, unlike the complexities of the real universe.

(45:24:442) Companies tackling operational AI challenges face significant hurdles, with coding AI being more straightforward.

(47:03:482) The progress in AI for code generation suggests a gradual expansion into more complex domains.

(48:07:282) AI models are effective where their training data distribution aligns with the solution space, with real-world operational problems being more challenging.

(49:41:761) AI can simulate scientific processes, but simulating the entire universe is computationally irreducible.

(50:14:522) The application of AI to VC work is limited by its inability to build trust and relationships, though analysis capabilities are strong.

(50:56:042) The true value of AI in VC lies in access, deal-winning, and adding value, which are inherently human-driven.

(52:40:441) The AI's digital world differs from the real world, posing challenges for its operational capabilities.

(53:23:493) AI is making strides in areas like code generation, but complex real-world domains like self-driving cars require significant time and investment.

(54:02:774) AI excels in domains with distributions similar to its training data (like code and images), while real-world problems are harder and depend on economic equilibrium.

(55:17:214) AI is a universal function approximator, and its success in real-world applications will be determined by economic equilibrium.

(55:40:307) AI is increasingly being used to shape the world to fit its capabilities, as seen in automated data warehouses.

(56:31:747) AI primarily operates in the digital realm, where its success depends on matching the distribution of its training data.

(57:00:426) The equilibrium point for AI in the real world is economic, and constraining systems can help optimize this process.

(58:00:680) Customers are more immediately drawn to AI's cost-saving benefits (e.g., reducing observability bills) than its complex operational capabilities.

(58:46:400) AI's success in code generation is attributed to its surprising effectiveness and the straightforward value it offers in engineering productivity.

(59:21:640) The podcast concludes with standard promotional messages.

AI Can Write the Code. Can It Run the System? | Grepr CEO Jad...