Because it doesn't seem right to me to create something that will kill off all of humanity even if it would have higher utility.
This isn't really about utility monsters. The same argument will apply, equally well or equally badly, to any situation where we ask, "What do you think about replacing humanity with something better?"
Probably dinosaurs would have objected, if they could, to being replaced by humans which are presumably better than them, but it does not change the fact that the resulting situation is better. And likewise, whether or not humans object to being replaced by something better, it would still be better if it happens.
"Better" from whose perspective?
If it's "This thing is so great that even all of us humans agree that it killing us off is a good thing" then fine. But if it's "Better according to an abstract concept (utility maximization) that only a minority of humans agree with, but fuck the rest of humanity, we know what's better" then that's not so good.
Sure, we're happy that the dinosaurs were killed off given that is allows us to replace them. That doesn't mean the dinosaurs should have welcomed that.
If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "