Superintelligence: Paths, Dangers, Strategies is a 2014 non-fiction book by Swedish philosopher Nick Bostrom.
Overview
The book argues that if machine brains surpass human brains in general intelligence, this new superintelligence could become extremely powerful and potentially pose an existential risk to humanity.
Key Concepts
- Intelligence explosion - The potential for AI to rapidly improve itself
- Orthogonality thesis - Intelligence and goals are independent; a superintelligent AI could have any goals
- Instrumental convergence - Certain sub-goals (self-preservation, resource acquisition) are useful for almost any final goal
- Control problem - How to ensure AI remains aligned with human values
AI Themes
- Existential risk from artificial general intelligence
- AI alignment and value loading
- The difficulty of specifying human values
- Containment strategies and their limitations
Influence
The book has been highly influential in the AI safety research community and has shaped public discourse on the potential risks of advanced AI systems.