Survey taken. I hope I didn't break it - I am a committed atheist, but also an active member of a Unitarian Universalist congregation, and I indicated that in spite of the explicit request for atheists not to answer the denomination question. (Atheist UUs are very common, and people on the "agnostic or less religious" side of the spectrum probably make up around 40% of the UU congregations I'm familiar with.)

What if the AI's utility function is to find the right utility function, being guided along the way? Its goals could be such as learning to understand us, obey us, and predict what we might want/like/approve, moving its object-level goals to what would satisfy humanity? In other words, a probabilistic utility function with great amounts of uncertainty, and great amounts of apprehension to change, or stability.

Regardless of the above questions/statement, I think much of the complexity of human utility comes from complexities of belief.

If we offload complexity of the AI's utility function into very uncertainly defined concepts, and give it an apprehension to do anything but observe given such little data... I don't know, though. This has been something I've been sitting on for a while, lambast me.

As one last thing, I think the best kind of FAI would be a singleton, with a metautility function, or society's utility function. I think one part of Friendliness would be determining a utility function for society, as to how people can interfere with each other in what circumstances, and then build the genie's utility function in the singleton's constraints.

Please critique. If my ideas are as unclear as I think they may be (I'm sick), please mention it.

Random thoughts:

1. The decision that smart high school students should take calculus rather than statistics (in the U.S.) strikes me as pretty seriously misguided. Statistics has broader uses.

2. I got through four semesters of engineering calculus; that was the clear limit of my abilities without engaging in the troublesome activity of "trying." I use virtually no calculus now, and would be fine if I forgot it all (and I'm nearly there). I think it gave me no or almost no advantages. One readthrough of Scarne on Gambling (as a 12-year-old) gave me more benefit than the entirety of my calculus education.

3. I ended up as the mathiest guy around in a non-math job. But it's really my facility with numbers that makes it; my wife (who has a master's degree in math) says what I am doing is arithmetic and not math, but very fast and accurate arithmetic skills strike me as very handy. (As a prosecutor, my facility with numbers comes as a surprise to expert witnesses. Sometimes, they are sad afterward.)

4. Anecdotally, math education may make people crazy or attract crazy people disproportionately. I think that pursuit of any topic aligns your brain to think in a way conducive to that topic.

My tentative conclusions are that advanced statistics has uses in understanding the world; other serious math is fun but probably not optimal use of time, unless it's really fun. "Really fun," has value. This conclusion is based on general observation, and is hardly scientific; I may well be wrong.

Also, IMO it took guts to bring this to the LW community. So kudos for that, too.

I expect this is wrong both as a model of the community and of Raemon's model of the community. Appearances don't matter, and psychological effects do.

Raemon, this is really great. As a lay leader of a Unitarian Universalist congregation, I love what you say about the importance of ritual -- it can be strongly affecting, and can motivate people to action they might not otherwise take. If we can construct rituals that inspire and invigorate, without misleading, then that is a win.

I'd suggest that when doing this kind of ritual, we should invite guests who are almost-but-not-quite in the rationalist camp. It can be a tool to attract new minds.

I will try to do a similar event at my church next year. We have quite a few atheists and fellow travelers, so I think it will work well. And maybe there will be further opportunities for rational ritual during the year -- other noteworthy astronomical events, perhaps. Or maybe when Nobels are announced.

Also, IMO it took guts to bring this to the LW community. So kudos for that, too.

I voted up on your post, Yvain, as you've presented some really good ideas here. Although it may seem like I'm totally missing your point by my response to your 3 scenarios, I assure you that I am well aware that my responses are of the "dodging the question" type which you are advocating against. I simply cannot resist to explore these 3 scenarios on their own.

Pascal's Wager

In all 3 scenarios, I would ask Omega further questions. But these being "least convenient world" scenarios, I suspect it'd be all "Sorry, can't answer that" and then fly away. And I'd call it a big jerk.

For Pascal Wager's specific scenario, I'd probably ask Omega "Really? Either God doesn't exist or everything the Catholics say is correct? Even the self-contradicting stuff?" And of course, he'd decline to answer and fly away.

So then I'd be stuck trying to decide whether God doesn't exist, or logic is incorrect (i.e. reality can be logically self inconsistent). I'm tempted to adopt Catholicism (for the same reason I would one-box on Newcomb: I want the rewards), but I'm not sure how my brain could handle a non-logical reality. So I really don't know what would happen here.

But let's say Omega additionally tells me that Catholicism is actually self-consistent, and I just misunderstood something about it, before flying away. In that case, I guess I'd start to study Catholicism. If my revised view of Catholicism has me believe that it does some rather cruel stuff (stone people for minor offenses, etc.) then I'd have to weight that against my desire to not suffer eternal torture.

I mean, eternal torture is pretty frickin' bad. I think in the end, I'd convert. And I'd also try to convert as many other people as possible, because I suspect I'd need to be cruel to fewer people if fewer people went against Christianity.

The God-Shaped Hole

To clarify your scenario, I'm guessing Omega explicitly tells me that I will be happier if I believe something untrue (i.e. God). I would probably reject God in this case, as Omega is implicitly confirming that God does not exist, and I do care about truth more than happiness. I've already experience this in other manners, so this is a much easier scenario for me to imagine.

Extreme Altruism

I don't think I can overcome this challenge. No matter how much I think about it, I find myself putting up semantic stop signs. In my "least convenient world", Omega tells me that Africa is so poverty stricken, and that my contribution would be so helpful, that I would be improving the lives of billions of people, in exchange for giving up all my wealth. While I might not donate all my money to save 10, I think I value billions of lives more than my own life. Do I value it more than my own happiness? This is an extremely painful question for me to think about, so I stop thinking about it.

"Okay", I say to Omega, "what if I only donate X percent of my money, and keep the rest for myself?" In one possible "least convenient world", Omega tells me that the charity is run by some nutcase whom, for whatever reason, will only accept an all-or-nothing deal. Well, when I phrase it like that, I feel like not donating anything, and blaming it on the nutcase. So suppose instead Omega tells me "There's some sort of principles of economy of scale which is too complicated for me to explain to you which basically means that your contribution will be wasted unless you contribute at least Y amount of dollars, which coincidentally just happens to be your total net worth." Again, I'm torn and find it difficult to come to a conclusion.

Alternative, I say to Omega "I'll just donate X percent of my money." Omega tells me "that's good, but it's not optimum." And I reply "Okay, but I don't have to do the optimum." but then Omega convinces me that actually, yes, I really should be doing the optimum somehow. Perhaps something along the line of how my current "ignore Africa altogether" behaviour is better than the behaviour of going to Africa and killing, torturing, raping everyone there. That doesn't mean that the "ignore Africa" strategy is moral.

I'm 19 years old at the University of Chicago, originally from Maryland. I'm a biochemistry major and physics minor, and have almost no idea which area of these field to go into. My only expressed goals for work are to do research somewhere and to have my work improve the overall quality of life by some amount.

I was raised in a Jewish household, but slowly turned atheist between the ages of 14 and 16. My parents were rarely around due to work, and they took little interest in my schoolwork, assuming I could handle it. As an only child, I was essentially raised by the internet. Honestly, I'm shocked I turned out as well as I did. I have little trouble with akrasia, as I use the method of procrastinating work with different work.

I'm called a morning person, but that's just because I always get almost exactly 7 hours of sleep every night. So, if I stay up until 2, I'll be awake by 9. Unless I get bored by a teacher's lecture, this method ensures that I'm almost never tired during the day but still able to fall asleep easily in the evening.

At the beginning of the year, I joined my school's circus club, which was surprisingly fun and fairly easy to pick up. I previously did glowstringing (poi) which is what got me into circus, but now I do various acrobatics, some stilt work, and a bit of juggling.

Additionally, I recently got a research position modeling protein folding that requires the use of unix and python. I'm pretty happy about this, as this will force me to actually learn a programming language, a mid-priority goal of mine for a while now.

Semi-rare poster. I was almost two-hundred years off. I think it might be the latin title that throws people.

I took the survey, sometime last week I think. EDIT: I think I may also have messed up the "two-digit probabilities" formatting requirement. I can't recall specifically any answer that might have violated it, but I also don't recall paying attention to that requirement while answering the survey.

Hello, all. I'm Joe. I'm 43, currently a graduate student in computational biology (in which I am discovering that a lot of inference techniques in biology are based on Bayes's Theorem). I'm also a professional software developer, and have been writing software for most of my life (since about age 10). In the early 1990's I was a graduate student at the AI lab at the University of Georgia, and though I didn't finish that degree, I learned a lot of stuff that was of great utility in my career in software development -- among other things, I learned about a number of different heuristics and their failure modes.

I remember a moment early in my professional career when I was trying to convince someone that some bug wasn't my fault, but was a bug in a third-party library. I very suddenly realized that, in fact, the problem was overwhelmingly more likely to be in my code than in the libraries and other tools we used, tools which were exercised daily by hundreds of thousands of developers. In that instant, I become much more skeptical of my own ability to do things Right. I think that moment was the start of my journey as a rationalist. I haven't thought about that process in a systematic way, though, until recently.

I've known of LW for quite a while, but really got interested when lukeprog of http://commonsenseatheism.com started reading Eliezer's posts sequentially. I'm now reading the sequences somewhat chaotically; I've read around 30% of the sequence posts.

My fear is, no matter how far I progress as a rationalist, I'll still be doing it Wrong. Or I'll still fear that I'm doing it wrong. I think I suffer greatly from under-confidence http://lesswrong.com/lw/c3/the_sin_of_underconfidence/ , and I'm very risk-averse. A property which I've just lately begun to view as a liability.

I am coming to view formal probabilistic reasoning as of fundamental importance to understanding reality, and I'd like to learn all I can about it.

If I overcome my reluctance to be judged by this community, I might write about my experiences with education in the US, which I believe ill-serves many of its clients. I have a 14-year-old daughter who is "unschooled". The topics of raising children as rationalists, and rational parenting, could engender some valuable discussions.

I might write about how, as an atheist, I've found it practically useful to belong to a religious community (a Unitarian Universalist church). "Believing in" religion is obviously irrational, but being connected with a religious community can in some circumstances be a rational, and non-cynical, move.

I might also write about software debugging as a rational activity. Though that's kind of obvious, I guess. OTOH debugging is IMO a severely under-valued skill in the field of software development. Most of my work is in soft real-time systems, which requires a whole different approach to debugging than interactive/GUI/web application development.

I might write about my own brief bout with mental illness, and about the process of dealing with a severely mentally-ill close relative, from a rationalist perspective.

My favorite sentence on LW so far: "Rationalists should WIN."