I feel that perhaps you are operating on a different definition of unpack than I am. For me, "can be good at everything" is less evocative than "achieves its value when presented with a wide array of environments" in that the latter immediately suggests quantification whereas the former uses qualitative language, which was the point of the original question as far as I could see. To be specific: Imagine a set of many different non-trivial agents all of whom are paper clip maximizers. You created copies of each and place them in a variety of non-trivial simulated environments. The ones that average more paperclips across all environments could be said to be more intelligent.

Have you seen the previous LW posts on the subject?

So I'm interested in taking up meditation, but I don't know how/where to start. Is there a practical guide for beginners somewhere that you would recommend?

I hate to sound negative

Somehow, I doubt this.

I predicted drops would fly off as the cloth was twisted. I was completely wrong.

I wrote an answer, but upon rereading, I'm not sure it's answering your particular doubts. It might though, so here:

Well, if we're talking about utilitarianism specifically, there are two sides to the answer. First, you favour the optimization-that-is-you more than others because you know for sure that it implements utilitarianism and others don't (thus having it around longer makes utilitarianism more likely to come to fruition). Basically the reason why Harry decides not to sacrifice himself in HPMoR. And second, you're right, there may well be a point where you should just sacrifice yourself for the greater good if you're a utilitarian, although that doesn't really have much to do with dissolution of personal identity.

But I think a better answer might be that:

If I have the choice, I might as well choose some other set of these moments, because as you said, "why not"?

You do not, in fact, have the choice. Or maybe you do, but it's not meaningfully different from deciding to care about some other person (or group of people) to the exclusion of yourself if you believe in personal identity, and there is no additional motivation for doing so. If you mean something similar to Eliezer writing "how do I know I won't be Britney +5 five seconds from now" in the original post, that question actually relies on a concept of personal identity and is undefined without it. There's not really a classical "you" that's "you" right now, and five seconds from now there will still be no "you" (although obviously there's still a bunch of molecules following some patterns, and we can assume they'll keep following similar patterns in five seconds, there's just no sense in which they could become Britney).

From an instrumental viewpoint, I hope you plan to figure out how to make everyone sitting around on a higher level credibly precommit to not messing with the power plug on your experience machine, otherwise it probably won't last very long. (Other than that, I see no problems with us not sharing some terminal values.)

(Unless you mind being simulated, in which case at least you'll never know.)

If I paid you to extend the lives of cute puppies, and instead you bought video games with that money but still sent me very convincing pictures of cute puppies that I had "saved", then you have still screwed me over. I wasn't paying for the experience of feeling that I had saved cute puppies -- I was paying for an increase in the probability of a world-state in which the cute puppies actually lived longer.

Tricking me into thinking that the utility of a world state that I inhabit is higher than it actually is isn't Friendly at all.

I actually don't think any of those things are problematic. A reductionist view of personal identity mainly feels like an ontology shift - you need to redefine all the terms in your utility function (or other decision-making system), but most outcomes will actually be the same (with the advantage that some decisions that were previously confusing should now be clear). Specifically:

Does the reductionist view of personal identity affect how we should ethically evaluate death?

I don't think so! You can redefine death as a particular (optionally animal-shaped) optimization process ceasing operation, which is not reliant on personal identity. (Throw in a more explicit reference to lack of continuity if you care about physical continuity.) The only side-effect of the reductionist view, I feel, is that it makes out preferences feel more arbitrary, but I think that's something you have to accept either way in the end.

For instance, if continuing to exist is like bringing new conscious selves into existence (by omission of not killing oneself), and if we consider continued existence ethically valuable, wouldn't this imply classical total utilitarianism, the view that we try to fill the universe with happy moments?

Not really. You can focus your utility function on one particular optimization process and its potential future execution, which may be appropriate given that the utility function defines the preference over outcomes of that optimization process.

Also, the idea of "living as long as possible" appears odd under this view, like an arbitrary grouping of certain future conscious moments one just happens to care about (for evolutionary reasons having nothing to do with "making the world a better place").

This is true enough. If you have strong preferences for the world outside of yourself (general "you"), you can argue that continuing the operation of the optimization process with these preferences increases the probability of the world more closely matching these preferences. If you care mostly about yourself, you have to bite the bullet and admit that that's very arbitrary. But since preferences are generally arbitrary, I don't see this as a problem.

Finally, in the comments someone remarked that he still has an aversion to creating repetitive conscious moments, but wouldn't the reductionist view on personal identity also undermine that? For *whom" would repetition be a problem?

This basically comes down to the fact that just because you believe that there's no continuity of personal identity, you don't have to go catatonic (or epileptic). You can still have preferences over what to do, because why not? The optimization process that is your body and brain continues to obey the laws of physics and optimize, even though the concept of "personal identity" doesn't mean much. (I'm really having a lot of trouble writing the preceding sentence in a clear and persuasive way, although I don't think that means it's incorrect.)

And in case someone thinks that I over-rely on the term "optimization process" and the comment would collapse if it's tabooed, I'm pretty sure that's not the case! The notion should be emergent as a pattern that allows more efficient modelling of the world (e.g. it's easier to consider a human's actions than the interaction of all particles that make up a human), and the comment should be robust to a reformulation along these lines.

This comment is thoroughly discouraging to me as it pokes at some open wounds that I'm working on.

Therefor I'm quitting LW for lent.

Kick my ass if I'm back before April fools day.

(except for meetup posts).