From Gleick's Chaos:
Every scientist who turned to chaos [theory] early had a story to tell of discouragement or open hostility. Graduate students were warned that their careers could be jeopardized if they wrote theses in an untested discipline, in which their advisors had no expertise. A particle physicist, hearing about this new mathematics, might begin playing with it on his own, thinking it was a beautiful thing, both beautiful and hard — but would feel that he could never tell his colleagues about it. Older professors felt they were suffering a kind of midlife crisis, gambling on a line of research that many colleagues were likely to misunderstand or resent...
Those who recognized chaos in the early days agonized over how to shape their thoughts and findings into publishable form. Work fell between disciplines — for example, too abstract for physicists yet too experimental for mathematicians. To some the difficulty of communicating the new ideas and the ferocious resistance from traditional quarters showed how revolutionary the new science was. Shallow ideas can be assimilated; ideas that require people to reorganize their picture of the world provoke hostility.
More (#3) from Chaos:
...Hubbard began using a computer to do what the orthodox techniques had not done. The computer would prove nothing. But at least it might unveil the truth so that a mathematician could know what it was he should try to prove. So Hubbard began to experiment. He treated Newton’s method not as a way of solving problems but as a problem in itself. Hubbard considered the simplest example of a degree-three polynomial, the equation x3– 1 =0. That is, find the cube root of 1. In real numbers, of course, there is just the trivial solution: 1. B
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: