(I am in the midst of reading the EY-RH "FOOM" debate, so some of the following may be less informed than would be ideal.)
From a purely technical standpoint, one problem is that if you permit self-modification, and give the baby AI enough insight into its own structure to make self-modification remotely a useful thing to do (as opposed to making baby repeatedly crash, burn, and restore from backup), then you cannot guarantee that utility() won't be modified in arbitrary ways. Even if you store the actual code implementing utility() in ROM, baby could self-modify to replace all references to that fixed function with references to a different (modifiable) one.
What you need is for utility() to be some kind of fixed point in utility-function space under whatever modification regime is permitted, or... something. This problem seems nigh-insoluble to me, at the moment. Even if you solve the theoretical problem of preserving those aspects of utility() that ensure Friendliness, a cosmic-ray hit might change a specific bit of memory and turn baby into a monster. (Though I suppose you could arrange, mathematically, for that particular possibility to be astronomically unlikely.)
I agree that basic probability and statistics is more practically useful than basic calculus, and should be taught at the high-school level or even earlier. Probability is fun and could usefully be introduced to elementary-school children, IMO.
However, more advanced probability and stats stuff often requires calculus. I have a BS in math and many years of experience in software development (IOW, not much math since college). I am in a graduate program in computational biology, which involves more advanced statistical methods than I'd been exposed to before, including practical Bayesian techniques. Calculus is used quite a lot, even in the definition of basic probabilistic concepts such as expectation of a random variable. Anything involving continuous probability distributions is going to be a lot more straightforward if approached from a calculus perspective. I, too, had four semesters of calculus as an undergrad and had forgotten most of it, but I found it necessary to refresh intensely in order to do well.
Noted. I'll try to update accordingly.
Raemon, this is really great. As a lay leader of a Unitarian Universalist congregation, I love what you say about the importance of ritual -- it can be strongly affecting, and can motivate people to action they might not otherwise take. If we can construct rituals that inspire and invigorate, without misleading, then that is a win.
I'd suggest that when doing this kind of ritual, we should invite guests who are almost-but-not-quite in the rationalist camp. It can be a tool to attract new minds.
I will try to do a similar event at my church next year. We have quite a few atheists and fellow travelers, so I think it will work well. And maybe there will be further opportunities for rational ritual during the year -- other noteworthy astronomical events, perhaps. Or maybe when Nobels are announced.
Also, IMO it took guts to bring this to the LW community. So kudos for that, too.
I mean, eternal torture is pretty frickin' bad. I think in the end, I'd convert. And I'd also try to convert as many other people as possible, because I suspect I'd need to be cruel to fewer people if fewer people went against Christianity.
This is a very good point, and I believe I'll point it out to my rather fundamentalist sibling when next we talk about this: if I really, truly believed that every non-Christian was doomed to eternal damnation, you can bet I'd be an evangelist!
Extreme Altruism
While I might not donate all my money to save 10, I think I value billions of lives more than my own life. Do I value it more than my own happiness? This is an extremely painful question for me to think about, so I stop thinking about it.
I definitely don't value those billions of lives more than my own happiness, or more than the happiness of those I know and love. However, I would seriously consider giving all of my wealth if Omega assured me that me and mine would be able to continue to be reasonably happy after doing so, even if it meant severe lifestyle changes.
I seem to be succeeding in helping to convince my graduate program in bioinformatics to ditch Perl in favor of Python. I'm very happy about this! When you don't have a programming background, and you're going into a field with heavy programming, Perl will hurt you -- it's likely to make you dislike programming. Python OTOH is like the fuzzy kitten of programming languages -- but it still has claws! (By which I mean, you can do serious stuff with it, despite its apparent adorableness.)
Also I've just started juggling again after a longish hiatus. I just decided to try a four-ball pattern the other day, and was absolutely shocked when I kept it going for like four complete cycles. Next mileposts will be: five-ball cascade, and three balls one-handed. I think 3/1 is probably harder than 5/2, but I'm not sure. I did a 3/1 flash the other day after ten tries, but I've never been able to complete a 5/2 flash. OTOH I've only recently begun to regard a 5-ball pattern as even achievable.
I was over 100 years off, but in the opposite direction.
I took the survey, sometime last week I think. EDIT: I think I may also have messed up the "two-digit probabilities" formatting requirement. I can't recall specifically any answer that might have violated it, but I also don't recall paying attention to that requirement while answering the survey.
Hello, all. I'm Joe. I'm 43, currently a graduate student in computational biology (in which I am discovering that a lot of inference techniques in biology are based on Bayes's Theorem). I'm also a professional software developer, and have been writing software for most of my life (since about age 10). In the early 1990's I was a graduate student at the AI lab at the University of Georgia, and though I didn't finish that degree, I learned a lot of stuff that was of great utility in my career in software development -- among other things, I learned about a number of different heuristics and their failure modes.
I remember a moment early in my professional career when I was trying to convince someone that some bug wasn't my fault, but was a bug in a third-party library. I very suddenly realized that, in fact, the problem was overwhelmingly more likely to be in my code than in the libraries and other tools we used, tools which were exercised daily by hundreds of thousands of developers. In that instant, I become much more skeptical of my own ability to do things Right. I think that moment was the start of my journey as a rationalist. I haven't thought about that process in a systematic way, though, until recently.
I've known of LW for quite a while, but really got interested when lukeprog of http://commonsenseatheism.com started reading Eliezer's posts sequentially. I'm now reading the sequences somewhat chaotically; I've read around 30% of the sequence posts.
My fear is, no matter how far I progress as a rationalist, I'll still be doing it Wrong. Or I'll still fear that I'm doing it wrong. I think I suffer greatly from under-confidence http://lesswrong.com/lw/c3/the_sin_of_underconfidence/ , and I'm very risk-averse. A property which I've just lately begun to view as a liability.
I am coming to view formal probabilistic reasoning as of fundamental importance to understanding reality, and I'd like to learn all I can about it.
If I overcome my reluctance to be judged by this community, I might write about my experiences with education in the US, which I believe ill-serves many of its clients. I have a 14-year-old daughter who is "unschooled". The topics of raising children as rationalists, and rational parenting, could engender some valuable discussions.
I might write about how, as an atheist, I've found it practically useful to belong to a religious community (a Unitarian Universalist church). "Believing in" religion is obviously irrational, but being connected with a religious community can in some circumstances be a rational, and non-cynical, move.
I might also write about software debugging as a rational activity. Though that's kind of obvious, I guess. OTOH debugging is IMO a severely under-valued skill in the field of software development. Most of my work is in soft real-time systems, which requires a whole different approach to debugging than interactive/GUI/web application development.
I might write about my own brief bout with mental illness, and about the process of dealing with a severely mentally-ill close relative, from a rationalist perspective.
My favorite sentence on LW so far: "Rationalists should WIN."
Survey taken. I hope I didn't break it - I am a committed atheist, but also an active member of a Unitarian Universalist congregation, and I indicated that in spite of the explicit request for atheists not to answer the denomination question. (Atheist UUs are very common, and people on the "agnostic or less religious" side of the spectrum probably make up around 40% of the UU congregations I'm familiar with.)