More (#1) from Chaos:
...for decades, Mandelbrot believes, he had to play games with his work. He had to couch original ideas in terms that would not give offense. He had to delete his visionary-sounding prefaces to get his articles published. When he wrote the first version of his book, published in French in 1975, he felt he was forced to pretend it contained nothing too startling. That was why he wrote the latest version explicitly as “a manifesto and a casebook.” He was coping with the politics of science.
“The politics affected the style in a sense which I later came to regret. I was saying, ‘It’s natural to…, It’s an interesting observation that….’ Now, in fact, it was anything but natural, and the interesting observation was in fact the result of very long investigations and search for proof and self-criticism. It had a philosophical and removed attitude which I felt was necessary to get it accepted. The politics was that, if I said I was proposing a radical departure, that would have been the end of the readers’ interest.
“Later on, I got back some such statements, people saying, ‘It is natural to observe…’ That was not what I had bargained for.”
Looking back, Mandelbrot saw that scientists in various disciplines responded to his approach in sadly predictable stages. The first stage was always the same: Who are you and why are you interested in our field? Second: How does it relate to what we have been doing, and why don’t you explain it on the basis of what we know? Third: Are you sure it’s standard mathematics? (Yes, I’m sure.) Then why don’t we know it? (Because it’s standard but very obscure.)
Mathematics differs from physics and other applied sciences in this respect. A branch of physics, once it becomes obsolete or unproductive, tends to be forever part of the past. It may be a historical curiosity, perhaps the source of some inspiration to a modern scientist, but dead physics is usually dead for good reason. Mathematics, by contrast, is full of channels and byways that seem to lead nowhere in one era and become major areas of study in another. The potential application of a piece of pure thought can never be predicted. That is why mathematicians value work in an aesthetic way, seeking elegance and beauty as artists do. It is also why Mandelbrot, in his antiquarian mode, came across so much good mathematics that was ready to be dusted off.
So the fourth stage was this: What do people in these branches of mathematics think about your work? (They don’t care, because it doesn’t add to the mathematics. In fact, they are surprised that their ideas represent nature.)
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: