Scientist AI
Develop powerful, nonagentic, uncertain world models that accelerate scientific progress while avoiding the risks of agent AIs
Theory of Change:Developing non-agentic 'Scientist AI' allows us to: (i) reap the benefits of AI progress while (ii) avoiding the inherent risks of agentic systems. These systems can also (iii) provide a useful guardrail to protect us from unsafe agentic AIs by double-checking actions they propose, and (iv) help us more safely build agentic superintelligent systems.
General Approach:Cognitive
Target Case:Pessimistic
Some names:Yoshua Bengio
Estimated FTEs:1-10
Critiques:
Hard to find, but see Raymond Douglas' comment, Karnofsky-Soares discussion. Perhaps also Predict-O-Matic.
Outputs:
Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?— Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, Marc-Antoine Rondeau, Pierre-Luc St-Charles, David Williams-King
The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems— Ziming Luo, Atoosa Kasirzadeh, Nihar B. Shah