Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

HungryHobo comments on A map: Causal structure of a global catastrophe - Less Wrong

4 Post author: turchin 21 November 2015 04:07PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (2)

You are viewing a single comment's thread.

Comment author: HungryHobo 24 November 2015 05:37:21PM 1 point [-]

Where's the arrow from "Creation of Strong benevolent AI" to "Agent becomes malicious". No matter how smart, your AI can make mistakes.

Also, I'm sure someone will point out that your chart makes a large number of assumptions and assumes that ours is the only answer.

Would you find this chart convincing if it was written by someone who believed that we should switch to a low-tech agrarian society like the amish and instead of everything in yellow had a chain down the side reading "reject technology", "turn to god" etc

Or instead of most of the boxes after the current day, a chain which reads something like "culture stagnates"->"technological progress slows"-> "technological progress regresses but not all the way"

Also your "roadmap" sets some mental alarm bells ringing since such massive nests of arrows rarely accompany good positions.

You've really got the LHC in there under "unintended consequences" leading to "agent becomes malicious". Really?

Comment author: turchin 26 November 2015 06:32:15PM 0 points [-]

"Where's the arrow from "Creation of Strong benevolent AI" to "Agent becomes malicious". No matter how smart, your AI can make mistakes." Yes it is true, and I have another map about risks of AI where this possibility is shown. I can't add too many arrows as the map is already too complex ))

"Also, I'm sure someone will point out that your chart makes a large number of assumptions and assumes that ours is the only answer.

Would you find this chart convincing if it was written by someone who believed that we should switch to a low-tech agrarian society like the amish and instead of everything in yellow had a chain down the side reading "reject technology", "turn to god" etc"

The part of this map which is dedicated to x-risks prevention deliberately made small as I have another map about x-risks prevention ways, where all known ideas about it presented in logical order. But it seems to me that your argument is more general, and basically you ask "Are you sure in what you are sure". I think that any map (or even article) may have a lot of assumptions, and it is impossible to list all of them in advance. Anyway it is not clear how to present assumptions in the map and keep it readable.

"You've really got the LHC in there under "unintended consequences" leading to "agent becomes malicious". Really?"

Yes. Unintended consequence of Large hadron collider may be creation of small black hole. The small black hole may become "malicious" if it starts to eat surrounding matter and grow exponentially. While term "malicious" may be strange in case of black hole, it in my opinion explains the difference between "simple" black hole and black hole that is able to eat matter. I think that to explain each arrow is needed a whole page of text (which I partly did in the book "Risks of human extinction" but its translation into English is still in draft.)