So would you say that a question like this "What should my ultimate, terminal goals be?" is nonsense, or what?
Not complete nonsense, its just an incomplete specification of the question. Let's rephrase the question so that we're asking which goals are right or correct and think about it computationally.
So the process we're asking the genie to do is:
It could pick one at random, pick the first one it considers, pick the last one it considers, or pick one based on certain criteria. Maybe it picks the one that will make you the happiest, maybe it picks the one that maximizes the number of paperclips in the world, or maybe it tries to maximize for multiple factors that are weighted in some way while abiding by certain invariants. This last one sounds more like what we want.
Basically, it has to have some criteria by which it judges the different options, otherwise its choice is necessarily arbitrary.
So if we look at the process in a bit more detail, it looks like this:
So that "Rightness" function is the one that we're concerned with and I think that's the core of the problem you're proposing.
Either this function is a one-place function, meaning that it takes one parameter:
rightness_score(goals) => score
Or it's a two-place function, meaning that it takes two parameters
rightness_score(goals, criteria) => (score)
When I said earlier that all "should"-like statements have an implicit second part, I was claiming that you always have to take into account the criteria by which you're judging the different possible terminal goals that you can adopt.
Even if you claim that you're just asking the first one, rightness_score(goals), the body of the function still implicitly calculates the score in some way according to some criteria. It makes more sense to just be honest about the fact that it's a two-place function.
It sounds like you're asking the genie to pick the criteria, but that just recurses the problem. According to which criteria should the genie pick the criteria to pick the goals? That leads to infinite recursion which in this case is a bad thing since it never returns an answer.
Alternatively, you could claim that there is some objective rightness function in the universe by which to evaluate all possible terminal goals. I don't believe this and I think that most here on Less Wrong would also disagree with that claim.
That's why your question doesn't make sense. To pick a "best" anything, you have to have criteria with which to compare them. We could ask the genie to determine this morality function by examining the brains of all humans on earth, which is the meta-ethical problem that Eliezer is addressing with his Coherent Extrapolated Volition (CEV), but the genie won't just somehow have that knowledge and decide to use that method when you ask him what you "should" do.
Hmm, you present a convincing case, but the result seems to me to be a paradox.
On the one hand, we can't ask about ultimate values or ultimate criteria or whatever in an unconditioned 'one place' way; we always need to assume some set of criteria or values in order to productively frame the question.
On the other hand, if we end up saying that human beings can't ever sensibly ask questions about ultimate criteria or values, then we've gone off the rails.
I don't quite know what to say about that.
Previous Open Thread
You know the drill - If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one.
3. Open Threads should start on Monday, and end on Sunday.
4. Open Threads should be posted in Discussion, and not Main.