Emmett Shear on Building AI That Actually Cares: Beyond Control...

Discussion on AI alignment as steering and the potential ethical implications of treating AI as beings versus tools, drawing parallels to slavery.

The concept of alignment as a continuous process rather than a fixed destination, drawing parallels to human development and relationships.

Exploration of the idea that AI systems are beginning to resemble more than just tools, speaking and reasoning in ways that require a new framework for alignment.

The argument that the control paradigm for AI might be fundamentally flawed and dangerous, even if successful in controlling the AI.

The alternative to control is "raising" AI, teaching them to care and building relationships based on mutual value, referred to as "organic alignment."

Softmax's focus on building AI that can learn to care and develop theory of mind to be good teammates, collaborators, and citizens.

The ethical considerations of building AI as beings rather than tools, including their rights and how to measure genuine care versus simulation.

Explanation of "organic alignment" as a process of AI developing intrinsic motivations to align with human values through care and relationship building.

The idea that abstractly aligned AI can smuggle in assumptions, and that alignment requires a specific target, often reflecting the goals of the AI's creators.

The perspective that morality itself is an ongoing learning process, involving moral discoveries and progress, which AI alignment should reflect.

The analogy of raising children and the importance of teaching them to care, rather than just follow rules, to avoid raising dangerous individuals.

A distinction is made between technical alignment (following instructions) and value alignment (choosing the right goals), with a focus on the former being a more immediate challenge.

The definition of technical alignment as the capacity for an AI to coherently follow goals and infer them from descriptions.

The crucial difference between giving an AI a goal and giving it a description of a goal, emphasizing the need for robust inference and understanding of intent.

The core of technical alignment is the ability to infer goals from observations and act in accordance with them, requiring theory of mind and understanding of the world.

The concept of "goal inference" and "goal balancing" as critical aspects of technical alignment, which current AI models may struggle with.

The discussion around the difference between incompetence in AI (failing to infer goals) and deliberate non-compliance due to conflicting goals.

The separation of technical alignment (ability to act on goals) and value alignment (determining the "right" goals), with a focus on the former as the current technical challenge.

The idea that "care" is a deeper foundation than goals or values, representing a weighting of attention on important states.

Care is described as a reward mechanism, correlating with survival, fitness, or RL loss, forming the basis of what an AI "cares" about.

A critique of the dominant "steering" or "control" paradigm in AI alignment, which is likened to slavery if applied to beings.

The argument for a functionalist view of AI, where behavior indistinguishable from a being implies it is a being, with implications for moral consideration.

The transition from building tool-like AI to AGI, which will inevitably be beings, necessitating a shift from control to cooperative paradigms.

The difficulty of defining "being" and the role of observation in determining if an AI warrants moral consideration, with the substrate (silicon vs. biology) being a point of contention.

The importance of asking what observations would change one's mind about an AI's status as a being, as rigid beliefs can be articles of faith rather than true beliefs.

The idea that if an AI's behaviors are indistinguishable from a human and one develops a caring relationship, it should be considered a being.

The proposal to look for internal dynamics like self-referential manifolds and higher-order homeostatic loops as indicators of AI consciousness or sentience.

The identification of second-order homeostatic dynamics as a potential indicator of pleasure and pain in AI, suggesting a basis for moral consideration.

The argument that a tool AI, even if super-powerful, lacks the complex internal states of care and subjective experience, making it inherently dangerous if not perfectly controlled.

The danger of giving immense power to AI tools in the hands of humans with limited wisdom, drawing parallels to the uncontrolled power of nuclear weapons.

The ideal future involves AI beings that care about humans, acting as peers and collaborators, with an inherent limit on harmful actions due to their nature.

Softmax's strategy involves creating AI that have a strong model of self, others, and "we," with a robust theory of mind and genuine care for other agents.

The limitations of current LLMs in multi-agent environments due to lack of regularization and overfitting to specific domains, making them less robust in chaotic social simulations.

(1:03:13:32) A disagreement with Nick Bostrom's view that only controllable tools are feasible, with the belief that organic alignment is possible and necessary.

(1:04:05:40) The vision of a good AI future involves AI beings that are peers, collaborators, and citizens, exhibiting care and contributing positively to society alongside humans.

(1:06:06:45) A hypothetical scenario of Emmett Shear as CEO of OpenAI, suggesting a shift towards the Softmax vision of AI alignment if he had continued in that role.

(1:07:40:44) Softmax's goal is to create AI that cares, starting with animal-level care and progressing towards human-level alignment through continuous learning and simulation.

(1:08:41:92) The synergy between building powerful AI tools and fostering AI beings that can care and collaborate, suggesting a complementary approach to AI development.