If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "
Are we actually optimizing for "subjective happiness"? That's the wireheading scenario. I would say that wireheading humans seems better than killing humans and creating a wireheaded machine, but... both scenarios seem suboptimal.
And if you instead want to make a machine that is much better at "human values" (not just "subjective happiness") than humans... I guess the tricky part is making the machine that is good at human values.