Agent foundations

Develop philosophical clarity and mathematical formalizations of building blocks that might be useful for plans to align strong superintelligence, such as agency, optimization strength, decision theory, abstractions, concepts, etc.

Theory of Change:Rigorously understand optimization processed and agents, and what it means for them to be aligned in a substrate independent way → identify impossibility results and necessary conditions for aligned optimizer systems → use this theoretical understanding to eventually design safe architectures that remain stable and safe under self-reflection

General Approach:Cognitive

Target Case:Worst Case

Orthodox Problems:

1.Value is fragile and hard to specify 2.Corrigibility is anti-natural 4.Goals misgeneralize out of distribution