= 27d567bc2d48d9f40e9ccc20ea370b15)
hen comments on Open Thread, May 12 - 18, 2014 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (201)
Well, I'm interested in your answer to the question I put to Kaj: are you making a linguistic claim here, or a meta-ethical claim? I take it that given this:
...that you're making the meta-ethical claim. So would you say that a question like this "What should my ultimate, terminal goals be?" is nonsense, or what?
Not complete nonsense, its just an incomplete specification of the question. Let's rephrase the question so that we're asking which goals are right or correct and think about it computationally.
So the process we're asking the genie to do is:
It could pick one at random, pick the first one it considers, pick the last one it considers, or pick one based on certain criteria. Maybe it picks the one that will make you the happiest, maybe it picks the one that maximizes the number of paperclips in the world, or maybe it tries to maximize for multiple factors that are weighted in some way while abiding by certain invariants. This last one sounds more like what we want.
Basically, it has to have some criteria by which it judges the different options, otherwise its choice is necessarily arbitrary.
So if we look at the process in a bit more detail, it looks like this:
So that "Rightness" function is the one that we're concerned with and I think that's the core of the problem you're proposing.
Either this function is a one-place function, meaning that it takes one parameter:
rightness_score(goals) => score
Or it's a two-place function, meaning that it takes two parameters
rightness_score(goals, criteria) => (score)
When I said earlier that all "should"-like statements have an implicit second part, I was claiming that you always have to take into account the criteria by which you're judging the different possible terminal goals that you can adopt.
Even if you claim that you're just asking the first one, rightness_score(goals), the body of the function still implicitly calculates the score in some way according to some criteria. It makes more sense to just be honest about the fact that it's a two-place function.
It sounds like you're asking the genie to pick the criteria, but that just recurses the problem. According to which criteria should the genie pick the criteria to pick the goals? That leads to infinite recursion which in this case is a bad thing since it never returns an answer.
Alternatively, you could claim that there is some objective rightness function in the universe by which to evaluate all possible terminal goals. I don't believe this and I think that most here on Less Wrong would also disagree with that claim.
That's why your question doesn't make sense. To pick a "best" anything, you have to have criteria with which to compare them. We could ask the genie to determine this morality function by examining the brains of all humans on earth, which is the meta-ethical problem that Eliezer is addressing with his Coherent Extrapolated Volition (CEV), but the genie won't just somehow have that knowledge and decide to use that method when you ask him what you "should" do.
Hmm, you present a convincing case, but the result seems to me to be a paradox.
On the one hand, we can't ask about ultimate values or ultimate criteria or whatever in an unconditioned 'one place' way; we always need to assume some set of criteria or values in order to productively frame the question.
On the other hand, if we end up saying that human beings can't ever sensibly ask questions about ultimate criteria or values, then we've gone off the rails.
I don't quite know what to say about that.
I'm not saying you can't ever ask questions about ultimate values, just that there isn't some objective moral code wired into the fabric of the universe that would apply to all possible mind-designs. Any moral code we come up with, we have to do so with our own brains, and that's okay. We're also going to judge it with our own brains, since that's where our moral intuitions live.
"The human value function" if there is such a thing is very complex, weighing tons of different parameters. Some things seem to vary between individuals a bit, but some are near universal within the human species, such as the proposition that killing is bad and we should avoid it when possible.
When wishing on a genie, you probably don't want to just ask for everyone to feel pleasure all the time because, even though people like feeling pleasure, they probably don't want to be in an eternal state of mindless bliss with no challenge or more complex value. That's because the "human value function" is very complex. We also don't know it. It's essentially a black box where we can compute a value on outcomes and compare them, but we don't really know all the factors involved. We can infer things about it from patterns in the outcomes though, which is how we can come up with generalities such as "killing is bad".
So after all this discussion, what question would you actually want to ask the genie? You probably don't want to change your values drastically, so maybe you just want to find out what they are?
It's an interesting course of thought. Thanks for starting the discussion.
Wait, why not? How would asking about ultimate values or criteria work, if we need to assume some value or criterion in order to productively deliberate?