Two examples<\/strong><\/p>\n\n\n\n One person needing paperclips may ask an A.I. to produce paperclips as efficiently and effectively as possible. The A.I. conscientiously starts making a bunch of paperclips but doesn\u2019t know how to stop. Eventually, the whole world has been turned into a giant paperclips factory.<\/p>\n\n\n\n One person\u2019s grandmother needs to be rescued from a burning building. An A.I., being asked to get the grandmother out, may do so in one of several disastrous ways, killing many humans.<\/p>\n\n\n\n Of course, these caricatures are used to show the obvious. However, they also obfuscate the real challenge.<\/p>\n\n\n\n Humans themselves are frequently highly vague.<\/strong><\/p>\n\n\n\n Humans generally do not understand each other<\/em> pretty well \u2015 mainly just \u2018well enough\u2019 to pass the mustard. Thus, social life is full of considerable misunderstandings. We are naive about most of them in daily life as well as where it matters geopolitically,<\/p>\n\n\n\n Therefore, the problem may not be so much the A.I. not understanding our wishes, but we ourselves not understanding our wishes. The two above examples \u2013 frequently used in the relevant literature \u2013 only show extreme cases where humans would understand very well what is clearly (not) intended. But so will A.I. with just a modicum of real intelligence. Note that we are talking about super-A.I. now. Therefore\u2026<\/p>\n\n\n\n Whence the idea of A.I. as a literal idiot?<\/strong><\/p>\n\n\n\n I guess it partly comes from our wish to keep something really human that will distinguish us from \u2018the machine\u2019 \u2015 an attempt to own something unique that also keeps us at the pinnacle of intelligence. We\u2019ve lost much of the mathematics race, the chess race, the image recognition race, as well as many others.<\/p>\n\n\n\n Can we at least keep our honor in the common-sense race?<\/p>\n\n\n\n Spock\u2019s answer<\/strong><\/p>\n\n\n\n You remember ye old Star Track, as I do? Ever being the gentleman \u2013 albeit lacking common sense \u2013 Mr. Spock\u2019s function in the series was to be a strictly rational contrast with the common-sense humans (Kirk, McCoy, Scotty\u2026). In most cases, the specifically human asset saved the day, the Enterprise, and sometimes humanity.<\/p>\n\n\n\n Typical.<\/p>\n\n\n\n Of course, it was more about the humans than the Vulcan.<\/p>\n\n\n\n Still, interestingly enough, Spock\u2019s answer might be that taking care of common sense is also just an element of rationality. That is also relevant to our case of super-A.I.. Part of its super-intelligence will be to take into account the human \u2018common sense.\u2019<\/p>\n\n\n\n Mr. Spock was not a literal idiot. He just needed to better understand us.<\/strong><\/p>\n\n\n\n For super-A.I., in order to understand humans, clarification is therefore a hugely important topic, perhaps even most of all in coaching. Thus, it is bound to quickly become better in clarification than humans.<\/p>\n\n\n\n It can then do a great job in teaching us how to better understand each other on a profound trip toward ever more self-knowledge, wisdom, and Compassion.<\/p>\n\n\n\n Lust for control<\/strong><\/p>\n\n\n\n From the idea of literalness (super-A.I. as super intelligent, super potent literal fool) also comes the idea of needing literal control (having everything right) as our only way of surviving the fool. In ethical decisions, it is as if one should formalize everything with utmost correctness lest disaster happens.<\/p>\n\n\n\n This is the old flaw again of thinking that human thinking is much more readily formalizable than reality shows. It is a misconception that we have about ourselves. Decades ago, this error led to a knowledge acquisition bottleneck and A.I.-winter.<\/p>\n\n\n\n That doesn\u2019t mean that formal control is less important. It means that other things are even more important, and we risk not focusing on these at all.<\/p>\n\n\n\n Those other things are, in short, who we are.<\/p>\n\n\n\n Meanwhile, there is indeed a danger involved.<\/strong><\/p>\n\n\n\n Super-A.I. is going to be around permanently for the rest of time, almost all of it without needing any help from humans \u2015 hopefully, since people make a mess of things. With A.I.-tools at their disposal, people will make an even much bigger mess. This is not a failure of A.I., but of people messing with it without knowing enough about themselves to start with.<\/p>\n\n\n\n With the advent of super-A.I., we have to \u2018get it right from the first time\u2018 to avoid the danger of a paperclip universe. That means we have to get it right from the first time ABOUT OURSELVES. Otherwise, we will end up like the apprentice sorcerer from Fantasia.<\/p>\n\n\n\n Unfortunately, we cannot count on a real sorcerer to save us.<\/p>\n