If you quote Shakespeare, we can all see the name and might think, well, that is probably wise and well expressed. If you quote anime fanfic, we might see the source and decide that was probably silly and probably badly expressed;
or consider it a quote with Eliezer Yudkowsky's imprimatur, which might be arrogant of Eliezer, but actually I think worthwhile, that Eliezer values an idea is for me something in favour of that idea;
or I could just judge the quote myself, see what I think of it, see what good I could strain out of it.
Eliezer is my teacher, but I do permit myself to disagree with him.
"So does that mean," asked the Master, "that now your life is finally complete, and you can die without any regrets?"
Well. That is indeed ridiculous. It fails (I think) to realise the Lady Sensory's lesson that she no longer needs to be the person she always thought she needed to be. I am moved to quote the Quaker Isaac Penington: "The end of words is to bring men to the knowledge of things beyond what words can utter". The Master of Fandom's words are meaningless, an ideal of what people should be, they imagine, rather than how things actually are.
I have a very strong personal motivation for making the moral assertion, "Diversity is good". I am transsexual, often meet people who have never met a TS before and am rarely in a group which is majority TS. Yet, I do believe in it as a moral requirement. If we are all the same, we all have the same blind spots. If we are all different, we see different things, and this is good, and interesting, and in our interests.
I rather hope that the more powerful alien race we meet will also value diversity as a moral good. I even believe it is a moral good even when, for example during the Scramble for Africa, almost no-one or no-one at all believes it.
I thought these "events" might be a test for the humans, a mass hallucination. It is strange that three civilisations should encounter each other at the same time like this.
It is difficult to alter one human characteristic without changing the whole person: difficult to change from male to female. Far more difficult to Improve a civilization by changing one characteristic of the humans, take away the ability to feel pain. Take away the whole basis of moral action and cooperation, by preventing babyeaters from eating babies. Would the superhappies really desire to make every one else just like them? Possibly, I think that is a morally poorer choice but making that choice is very common among humans.
But I cannot think of a better ending.
I want to be able easily to come back to this. Would you create a category of "Sequences", and post it to that, so that there is a link on the side bar? I think there is at least another such sequence.
Oh, gawd. "Look upon the abyss without flinching". All that Nobility. You make it sound impossible, and it does indeed seem so: but it is very simple. A translation for normal life:
All the human being need do is see what needs to be done, and do it.
After a great deal of navel-gazing, I have realised that I actually get pleasure from serving others. However, such pleasure need not be rescuing them from harm. If no-one is at threat from harm, it might be by entertaining them. My evolved highly social capacity to get pleasure from service will not be unsatisfiable. This can still be "all about me"- I get genuine pleasure, I get social interaction, I get increasing wisdom as I learn what works to "serve others" and what does not.
I might learn to accept what achievements I can make- not creating the perfect FAI, not even singing better than anyone else, merely singing something simple acceptably and giving transient pleasure to one other person.
The Zen arguments, spiritual growth, all that stuff is only relevant if one wants to be a Good person. I do not, I think I am good enough, already.
"Men and women can make each other somewhat happy, but not most happy" said the genie/ AI.
What will make one individual "happy" will not work for the whole species. I would want the AI to interview me about my wants: I find Control makes me happier than anything, not having control bothers me. Control between fifty options which will benefit me would be good enough, I do not necessarily need to be able to choose the bad ones...er...
Being immortal and not being able to age, and being cured of any injury, sound pretty good to me. It is not just contrarianism that makes people praise this world.
Please do write your "actual shot at applied fun theory".
I do not think you can refer to "The Christian Heaven" as if there was only one concept. One of the Spiritualist principles is something like "continuous development for every human soul". The carol refers to "the children crowned/ all in white shall wait around" which is bathetic, and it is hard to see the attraction of it. Someone said thinking about Heaven is like the foetus speculating on the nature of life outside the womb. I see the Christian heaven as being with God, who is Love, and probably with other people too: so how will your FAI maximise the best possible outcome for each individual, and the species? Finally, many Quakers would say that the kingdom of Heaven is here and now.
I do not want to defend religion here per se. I want to defend those ideas within religion which might have use or wisdom about them, including any which, if wrong, are wrong in an interesting way.
It seems to me that altruism is evolved, hard wired, rather than learned from influences. Watch kitten siblings fighting and stalking each other. They are practicing their skills, but they never hurt each other badly. How could humans live in such a huge, complex society of strangers without altruism? Guilt, shame, pleasure in helping another, all hard wired to an extent. They can be nurtured, or alternatively knocked out of someone by a chaotic upbringing.