ŷ������

Superintelligence was published in 2014, and it's already had time to become a cult classic. So, with apologies for being late getting to the party, here's my two cents.

For people who still haven't heard of it, the book is intended as a serious, hard-headed examination of the risks associated with the likely arrival, in the short- to medium-term future, of machines which are significantly smarter than we are. Bostrom is well qualified to do this. He runs the Future of Humanity Institute at Oxford, where he's also a professor at the philosophy department, he's read a great deal of relevant background, and he knows everyone. The cover quotes approving murmurs from the likes of Bill Gates, Elon Musk, Martin Rees and Stuart Russell, co-author of the world's leading AI textbook; people thanked in the acknowledgements include Demis Hassabis, the founder and CEO of Google's Deep Mind. So, why don't we assume for now that Bostrom passes the background check and deserves to be taken seriously. What's he saying?

First of all, let's review the reasons why this is a big deal. If machines can get to the point where they're even a little bit smarter than we are, they'll soon be a whole lot smarter than we are. Machines can think much faster than humans (our brains are not well optimised for speed); the differential is at least in the thousands and more likely in the millions. So, having caught us up, they will rapidly overtake us, since they're living thousands or millions of their years for every one of ours. Of course, you can still, if you want, argue that it's a theoretical extrapolation, it won't happen any time soon, etc. But the evidence suggests the opposite. The list of things machines do roughly as well as humans is now very long, and there are quite a few things, things we humans once prided ourselves on being good at, that they do much better. More about that shortly.

So if we can produce an artificial human-level intelligence, we'll shortly after have an artificial superintelligence. What does "shortly after" mean? Obviously, no one knows, which is the "fast takeoff/slow takeoff" dichotomy that keeps turning up in the book. But probably "slow takeoff" will be at most a year or two, and fast takeoff could be seconds. Suddenly, we're sharing our planet with a being who's vastly smarter than we are. Bostrom goes to some trouble to help you understand what "vastly smarter" means. We're not talking Einstein versus a normal person, or even Einstein versus a mentally subnormal person. We're talking human being versus a mouse. It seems reasonable to assume the superintelligence will quickly learn to do all the things a very smart person can do, including, for starters: formulating and carrying out complex strategic plans; making money in business activities; building machines, including robots and weapons; using language well enough to persuade people to do dumb things; etc etc. It will also be able to do things that we not only can't do, but haven't even thought of doing.

And so we come to the first key question: having produced your superintelligence, how do you keep it under control, given that you're a mouse and it's a human being? The book examines this in great detail, coming up with any number of bizarre and ingenious schemes. But the bottom line is that no matter how foolproof your scheme might appear to you, there's absolutely no way you can be sure it'll work against an agent who's so much smarter. There's only one possible strategy which might have a chance of working, and that's to design your superintelligence so that it wants to act in your best interests, and has no possibility of circumventing the rules of its construction to change its behavior, build another superintelligence which changes its behavior, etc. It has to sincerely and honestly want to do what's best for you. Of course, this is Asimov Three Laws territory; and, as Bostrom says, you read Asimov's stories and you see how extremely difficult it is to formulate clear rules which specify what it means to act in people's best interests.

So the second key question is: how do you build an agent which of its own accord wants to do "the right thing", or, as Socrates put it two and half thousand years ago, is virtuous? As Socrates concludes, for example in Meno and Euthyphro, these issues are really quite difficult to understand. Bostrom uses language which is a bit less poetic and a bit more mathematical, but he comes to pretty much the same conclusions. No one has much idea yet of how to do it. The book reaches this point and gives some closing advice. There are many details, but the bottom line is unsurprising given what's gone before: be very, very careful, because this stuff is incredibly dangerous and we don't know how to address the critical issues.

I think some people have problems with Superintelligence due to the fact that Bostrom has a few slightly odd beliefs (he's convinced that we can easily colonize the whole universe, and he thinks simulations are just as real as the things they are simulating). I don't see that these issues really affect the main arguments very much, so don't let them bother you if you don't like them. Also, I'm guessing some other people dislike the style, which is also slightly odd: it's sort of management-speak with a lot of philosophy and AI terminology added, and because it's philosophy there are many weird thought-experiments which often come across as being a bit like science-fiction. Guys, relax. Philosophers have been doing thought-experiments at least since Plato. It's perfectly normal. You just have to read them in the right way. And so, to conclude, let's look at Plato again (remember, all philosophy is no more than footnotes to Plato), and recall the argument from the Theaetetus. Whatever high-falutin' claims it makes, science is only opinions. Good opinions will agree with new facts that turn up later, and bad opinions will not. We've had three and a half years of new facts to look at since Superintelligence was published. How's its scorecard?

Well, I am afraid to say that it's looking depressingly good. Early on in the history of AI, as the book reminds us, people said that a machine which could play grandmaster level chess would be most of the way to being a real intelligent agent. So IBM's team built Deep Blue, which beat Garry Kasparov in 1997, and people immediately said chess wasn't a fair test, you could crack it with brute force. Go was the real challenge, since it required understanding. In late 2016 and mid 2017, Deep Mind's AlphaGo won matches against two of the world's three best Go players. That was also discounted as not a fair test: AlphaGo was trained on millions of moves of top Go matches, so it was just spotting patterns. Then late last year, Alpha Zero learned Go, Chess and Shogi on its own, in a couple of days, using the same general learning method and with no human examples to train from. It played all three games not just better than any human, but better than all previous human-derived software. Looking at the published games, any strong chess or Go player can see that it has worked out a vast array of complex strategic and tactical principles. It's no longer a question of "does it really understand what it's doing". It obviously understands these very difficult games much better than even the top experts do, after just a few hours of study.

Humanity, I think that was our final warning. Come up with more excuses if you like, but it's not smart. And read Superintelligence.