Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Anna_Salamon_and_Steve_Rayhawk comments on The Baby-Eating Aliens (1/8) - Less Wrong

42 Post author: Eliezer_Yudkowsky 30 January 2009 12:07PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (84)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Anna_Salamon_and_Steve_Rayhawk 01 February 2009 11:06:00AM 0 points [-]

Unknown, how certain are you that you would retain that preference if you "knew more, thought faster"? How certain are you that Eliezer would retain the opposite preference and that we are looking at real divergence? I have little faith in my initial impressions concerning Babyeaters vs. black holes; it's hard for me to understand the Babyeater suffering, or the richness of their lives vs. of black holes, as more than a statistic.

Eliezer, regarding (2), it seems plausible to me (I'd assign perhaps 10% probability mass) that if there is a well-formed goal that with the non-arbitrariness property that both Hollerith and Roko seem partly to be after, there is a reasonable notion of extrapolation (though probably a minority of such notions) under which 95% of humans would converge to that goal. Yes, Hollerith got there by a low-probability path; but the non-arbitrariness he is (sort of) aiming for, if realizable, suggests his aim could be gotten to by other paths as well. And there are variables in one's choice of "reasonable" notions of extrapolation that could be chosen to make non-arbitrariness more plausible. For example, one could give more weight to less arbitrary preferences (e.g., to whatever human tendencies lead us to appreciate Go or the integers or other parts of mathematics), or to types of value-shifts that to make our values less arbitrary (e.g., to preferences for values with deep coherence, or to preferences similar to a revulsion for lost purposes), or one could include father-back physical processes (e.g., biological evolution, gamma rays) as part of the "person" one is extrapolating.
[I realize the above claim differs from my original (2).] Do you disagree?

Richard, I don't see why pulling numbers out of the air is absurd. We're all taking action in the face of uncertainty. If we put numbers on our uncertainty, we give others more opportunity to point out problems in our models so we can learn (e.g., it's easier to notice if we're assigning too-high probabilities to conjunctions and so having probabilities that sum to more than 1).