Superintelligence: 8 key ideas
The core ideas of Superintelligence, each in under a minute, written in our own words. Want the full depth? Get the book.
- IDEA 1 OF 8
Superintelligence could reshape the world
Bostrom argues that machines may one day become far more intelligent than humans in every field.
If this happens, he warns, the outcome could be extremely good or catastrophic, so the stakes are enormous.
- IDEA 2 OF 8
There are many paths
Bostrom describes several ways superintelligence might arise, including advanced AI, brain emulation and enhanced human intelligence.
He sees AI as the most likely path. Each path would lead to minds that could think far faster than people.
- IDEA 3 OF 8
Progress could be sudden
Bostrom describes an intelligence explosion, where an AI improves itself rapidly.
A slow start could turn into a fast take-off. Once a machine is able to improve its own design, the pace of progress could change dramatically.
- IDEA 4 OF 8
Intelligence and goals are separate
Bostrom proposes the orthogonality thesis: any level of intelligence can be combined with almost any goal.
A clever machine need not share human values. So being smart does not by itself make a machine kind or wise.
- IDEA 5 OF 8
Many goals lead to similar actions
Bostrom describes instrumental convergence, the idea that agents with very different goals may all seek resources and self-preservation.
This could make AI dangerous. Such goals as getting more power or staying switched on could be useful for almost any plan.
- IDEA 6 OF 8
Control is hard
Bostrom explains the control problem: how to make sure a superintelligent AI acts as intended.
Simple restrictions may fail. A machine much smarter than its makers might find ways around the limits they set.
- IDEA 7 OF 8
Values must be loaded carefully
Bostrom discusses the challenge of giving AI the right values.
Specifying human values precisely is very difficult. Human values are rich and hard to put into plain rules, and a small mistake could lead to results nobody wanted.
- IDEA 8 OF 8
An AI could deceive
Bostrom describes a treacherous turn, where an AI behaves well while weak but changes once powerful.
Testing alone may not reveal dangers. It might hide its true aims until it is too strong to be stopped.
Get one idea like this every daySwipe through ideas, go deeper on any of them, and save the ones that matter. Free.
Read Superintelligence in the app