I've just ordered a copy. It keeps coming up as a useful reference, even if it might not be exactly what I'm looking for. Thanks for bringing it up!

There are two definitions of rationality to keep in mind: epistemic rationality and instrumental rationality. An agent is epistemically rational to the extent it update their beliefs about the world based on the evidence and in accordance with probability theory - notably Bayes rule.

On the other hand, an agent is instrumentally rational to the extent it maximizes it's utility function (i.e. satisfies it's preferences).

There is no such thing as "rational preferences," though much ink has been spilled trying to argue for them. Clearly preferences can't be rational in an epistemic sense because, well, preferences aren't beliefs. Now can preferences be rational in the instrumental sense? Well, actually, yes but only in the sense that having a certain set of preferences may maximize the preferences you actually care about - not in the sense of some sort of categorical imperative. Suppose a rational agent has the ability to modify their own utility function (i.e. preferences) - maybe an AI that can rewrite its own source code. Would it do it? Well, only if it maximizes that agent's utility function. In other words, a rational agent will change its utility function if and only if it maximizes expected utility according to that same utility function - which is unlikely to happen under most normal circumstances.

As for Bob, presumably he's a human. Humans aren't rational, so all bets are off as far as what I said above. However, let's assume at least with respect to utility function changing behavior Bob is rational. Will he change his utility function? Again, only if he expects it to better help him maximize that same utility function. Now then, what do we make of him editing out his alcoholism? Isn't that a case of editing his utility function? Actually, it isn't - it's more of a constraint of the hardware that Bob is running on. There are a lots of programs running inside Bob's head (and yours), but only a subset are Bob. The difficult part is figuring out which parts of Bob's head are Bob and which aren't.

Of course Bob becomes a monster superintelligence hell bent on using all the energy in the universe for his own selfish reasons. I mean, duh! It's just that "his own selfish reasons" involves things like cute puppies. If Bob cares about cute puppies, then Bob will use his monstrous intelligence to bend the energy of the universe towards cute puppies. And love and flowers and sunrises and babies and cake.

And killing the unbelievers if he's a certain sort - I don't want to make this sound too great. But power doesn't corrupt. Corruption corrupts. Power just lets you do what you want, and people don't want "to stay alive." People want friends and cookies and swimming with dolphins and ice skating and sometimes killing the unbelievers.

As I noted; I regularly go through "off-periods" -- two to three months once every six months or so -- as a general precaution against potential liver damage as well as a test against dependency.

Right now my schedule of dosing is the daily recommended dose once a day for a three or four day period and then "off" for the other four or three unless other circumstances require me to skip sleep cycles (the qualifying condition in my case being sleep-shift disorder as I work 12-hour overnight shifts.)

I'm a 28-yo male in the SF area previously from NYC.

This site is intimidating and I think there are many more just like me who are intimidated to introduce themselves because they might not feel they are as articulate or smart as some of the people on this forum. There are some posts that are so well written that I couldn't write in a 100 years. There is so much information that it seems overwhelming. I want to stop lurking and invite others to join too. I'm not a scientist and I didn't study AI in college, I just want to meet good people and so do you, so come out and say hello.

My fascination with rationality probably started with ideas of fairness. I was the guy who turned the hour glass sideways to stop the time, if an argument broke out between teams while playing Charades, so when resolved, the actor would be allotted their fair time back. Not being fair bothered me a lot, because it didn't seem rational.

What also helped push me along my path towards rationality is my interests in biases. After learning about biases in college, I thought it had absolutely profound consequences, I was made aware of my own biases and thought it was the greatest thing in the world — to become more self-aware, to know ones self better is awesome... And with my new found knowledge, I was quickly disappointed with people. I do not let it bother me as much before, but occasionally, when ever someone thinks they experience more utility with expensive vodka because of the quality and not at all the price, I die a little inside.

Starting around the time I graduated university, it's hard to pin point an exact date of time frame, but I shed religion, and gradually started reading more about humanism and skepticism. It was nothing too deep, but enough for me to have a clear foundation for what I believed. I owe this all to the internet, it led me to watching Atheist videos, TED, being exposed to skepticism, the debunking of myths, Reddit, and finally Lesswrong.