Indeed. Like many others here, I subscribe to emotivism as well as utilitarianism.

checking out the wikipedia article... hmm... I think I agree with emotivism too, to some degree. I already have a habit of saying "but that's just my opinion", and being uncertain enough about the validity (validity according to what?) of my preferences, to not dare to enforce them if other people disagree. And emotivism seems like a formalization of the "but that's just my opinion". That could be useful.

Yes, that's the ideal. But the planning fallacy tells us how much harder it is to make things work in practice than to imagine how they should work. Actually implementing CEV requires work, not magic, and that's precisely why we're having this conversation, as well as why SIAI's research is so important. :)

good point. and yeah, that's that's one of the main issues that's causing me to doubt whether SIAI has any hope of achieving their mission.

I hope so. Of course, it's not as though the only two possibilities are "CEV" or "extinction." There are lots of third possibilities for how the power politics of the future will play out (indeed, CEV seems exceedingly quixotic by comparison with many other political "realist" scenarios I can imagine), and having a broader base of memetic support is an important component of succeeding in those political battles. More wild-animal supporters also means more people with economic and intellectual clout.

good point. Have you had any contact with Metafire yet? He strongly agrees with you on this. Just recently he started posting to LW.

oh, and "quixotic", that's the word I was looking for, thanks :)

If you include paperclippers or suffering-maximizers in your definition of "anyone," then I'd put the probability close to 0%. If "anyone" just includes humans, I'd still put it less than, say, 10^-3.

heh, yeah, that "significantly less than 50%" was actually meant as an extremely sarcastic understatement. I need to learn how to express stuff like this more clearly.

Yeah, although if we take the perspective that individuals are different people over time (a "person" is just an observer-moment, not the entire set of observer-moments of an organism), then any choice at one instant for pain in another instant amounts to "forcing someone" to feel pain....

good point! This suggests the possibility of requiring people to go through regular mental health checkups after the Singularity. Preferably as unobtrusively as possible. Giving them a chance to release themselves from any restrictions they tried to place on their future selves. Though the question of what qualifies as "mentally healthy" is... complex and controversial.