Scary AI Is More “Fantasia” Than “Terminator”
When Nate Soares psychoanalyzes himself, he sounds less Freudian than Spockian. As a boy, he’d see people acting in ways he never would “unless I was acting maliciously,” the former Google software engineer, who now heads the non-profit Machine Intelligence Research Institute, reflected in a blog post last year. “I would automatically, on a gut level, assume that the other person must be malicious.” It’s a habit anyone who’s read or heard David Foster Wallace’s “This is Water” speech will recognize.
Later Soares realized this folly when his “models of other people” became “sufficiently diverse”—which isn’t to say they’re foolproof, he wrote in the same post. “I’m probably still prone to occasionally believing someone is malicious when they’re merely different than me, especially in cases where they act similarly to me most of the time (thereby fooling my gut-level person-modeler into modeling them too much like myself).” He suspected that this “failure mode” links up with the “typical mind” fallacy, and is “therefore,” he concluded, “difficult to beat in general.”
Beating biases is one of Soares’ core concerns—“I care a lot about having accurate beliefs,” he says—and one of Neil deGrasse Tyson that keeping a super intelligent machine “in a box,” disconnected from the Internet, was a lousy safeguard for a misaligned AI and Steven Pinker’s misconceptions about AI safety. Donors find the Institute’s mission worthwhile: In December 2017, 341 of them it $2.5 million.
