gwern comments on Common sense as a prior - Less Wrong

33 Post author: Nick_Beckstead 11 August 2013 06:18PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (212)

You are viewing a single comment's thread. Show more comments above.

Comment author: JonahSinick 12 August 2013 04:42:18AM 2 points [-]

Hi Brian :-)

For example, the reason most elites don't seem to take seriously efforts like shaping trajectories for strong AI is not because they think the probabilities of making a difference are astronomically small but because they don't bite Pascalian bullets.

How do you know this? It's true that their utility functions aren't linear, but it doesn't follow that that's why they don't take such efforts seriously. Near-Earth Objects: Finding Them Before They Find Us reports on concerted efforts to prevent extinction-level asteroids from colliding into earth. This shows that people are (sometimes) willing to act on small probabilities of human extinction.

Insofar as my own actions are atypical, I intend for it to result from atypical moral beliefs rather than atypical factual beliefs. (If you can think of instances of clearly atypical factual beliefs on my part, let me know.)

Dovetailing from my comment above, I think that there's a risk of following the line of thought "I'm doing X because it fulfills certain values that I have. Other people don't have these values. So the fact that they don't engage in X, and don't think that doing X is a good idea, isn't evidence against X being a good idea for me" without considering the possibility that despite the fact that they don't have your values, doing X or something analogous to X would fulfill their (different) values conditioning on your factual beliefs being right, so that the fact that they don't do or endorse X is evidence against your factual beliefs connected with X. In a given instance, there will be a subtle judgment call as to how much weight to give to this possibility, but I think that it should always be considered.

Comment author: gwern 19 August 2013 02:31:50AM 1 point [-]

This shows that people are (sometimes) willing to act on small probabilities of human extinction.

Unfortunately, it's a mixed case: there were motives besides pure altruism/self-interest. For example, Edward Teller was an advocate of asteroid defense... no doubt in part because it was a great excuse for using atomic bombs and keeping space and laser-related research going.