This is like posting an article about genetic evolution on a creationist forum. They will pretend to "understand" what you are saying and then dig down deeper into their irrational dogma.

Bostrom is a crypto-creationist "philosopher" with farcical arguments in favor of abrahamic mythology and neo-luddism. People are giving too much credit to lunatics who promote AI eschatology. Please do not listen to schizophrenics like Bostrom. The whole "academic career" of Bostrom may be summarized as "non-solutions to non-problems". I have never seen a less useful thinker. He could not be more wrong! I sometimes think that philosophy departments should be shut down, if they are to breed this kind of ignorance.

It's quite ironic that Bostrom is talking about superintelligence, BTW. How will he imagine what intelligent entities think?

Because life isn't a third grade science fiction movie, where the super scientists who program AI agents are at the same time so incompetent that their experiments break out of the lab and kill everyone.

This seems to be closer to an argument from ridicule than an argument with content. No one has said anything about "super scientists"- I am however mildly curious if you are familiar with the AI Box experiment. Are you claiming that AI aren't going to get to be effectively powerful or are you claiming that you inherently trust that safeguards will be sufficient? Note that these are not the same thing.

This misses the basic problem: Most of the ways things can go seriously wrong are things that would occur after the AGI is already an AGI and are things where once they've happened one cannot recover.

More concretely, what experiment in your view should they be doing?

Transcript:

Question: Are you as afraid of artificial intelligence as your Paypal colleague Elon Musk?

Thiel: I'm super pro-technology in all its forms. I do think that if AI happened, it would be a very strange thing. Generalized artificial intelligence. People always frame it as an economic question, it'll take people's jobs, it'll replace people's jobs, but I think it's much more of a political question. It would be like aliens landing on this planet, and the first question we ask wouldn't be what does this mean for the economy, it would be are they friendly, are they unfriendly? And so I do think the development of AI would be very strange. For a whole set of reasons, I think it's unlikely to happen any time soon, so I don't worry about it as much, but it's one of these tail risk things, and it's probably the one area of technology that I think would be worrisome, because I don't think we have a clue as to how to make it friendly or not.

Do you have any serious counter arguments to ideas presented in a Bostrom's book? Majority of top AI experts agree that we will have human-level AI by the end of this century, and people like Musk, Bostrom and MIRI guys are just trying to think about possible negative impacts that this development may have on humans. The problem is that the fate of humanity may depend on action of non-human actors, who will likely have utility function incompatible with human survival and it is perfectly rational to be worried about that.

Those ideas are definitely not above criticism but also should not be dismissed based on perceived lack of expertise. Someone like Elon Musk has actually direct contact with people who are working on one of the most advanced AI projects on earth (Vicarious, DeepMind), so he certainly knows what he is talking about.

+1 for entertainment value.

EDIT: I am not agreeing with examachine's comment, I just think it's hilariously bad.

achieving better than human performance on a sufficiently wide benchmark

You understand that you just replaced some words with others without clarifying anything, right? "Sufficiently wide" doesn't mean anything.

Racists. Why does anyone even care about such people? Just ignore them.

Well, his comment was deleted, possibly by him, so we should take that into account - maybe he thought he was being a bit overly Cassandra-like too.

The other thing to remember is that Musk's comments reach a slightly different audience to the usual with regards to AI risk. So it's at least somewhat relevant to see the perspective of the one communicating to these people.