"All rational beings will be moral, but this paper worries me that AI, while efficient, may not end up being rational. Maybe it's worth worrying about."

Why not argue for this directly, instead of making a much stronger claim ("may not" vs "very unlikely")? If you make a claim that's too strong, that might lead people to dismiss you instead of thinking that a weaker version of the claim could still be valid. Or they could notice holes in your claimed position and be too busy trying to think of attacks to have the thoughts that you're hoping for.

(But take this advice with a big grain of salt since I have little idea how academic philosophy works in practice.)

Thanks for this explanation. Does your OH participate in discussion here? If so does he enjoy them (more than discussions with "regular smart people")? Do you or he have any suggestions how we might better attract people like him (i.e., who are "naturally" rational and find it hard to understand at first why Eliezer is making a big deal out of "rationality")?

I do worry that thought experiments involving Omega can lead decision theory research down wrong paths (for example by giving people misleading intuitions), and try to make sure the ones I create or pay attention to are not just arbitrary thought experiments but illuminate some aspect of real world decision making problems that we (or an AI) might face. Unfortunately, this relies largely on intuition and it's hard to say what exactly is a valid or useful thought experiment and what isn't, except maybe in retrospect.

My conditional was "cautionary position is the correct one". I meant, provably correct.

Leaving out the "provably" makes a big difference. If you add "provably" then I think the conditional is so unlikely that I don't know why you'd assume it.

Ok, I guess I could be wrong then. Maybe somebody who knows Roko could ask him?

And done by mathematicians finding actual theorems, not by philosophers assuming there is an actual thing behind our use of the word, that it is their task to discover.

I don't mean to pick just on you, but I think philosophy is often unfairly criticized for being less productive than other fields, when the problem is just that philosophy is damned hard, and whenever we do discover, via philosophy, some good method for solving a particular class of problems, then people no longer consider that class of problems to belong to the realm of philosophy, and forget that philosophy is what allowed us to get started in the first place. For example, without philosophy, how would one have known that proving theorems using logic might be a good way to understand things like circles, lines, and shapes (or even came up with the idea of "logic")?

(Which isn't to say that there might not be wrong ways to do philosophy. I just think we should cut philosophers some slack for doing things that turn out to be unproductive in retrospect, and appreciate more the genuine progress they have made.)

I'm surprised and disconcerted that some people might be so afraid of being rebuked by Eliezer as to be reluctant to criticize/correct him even when such incontrovertible evidence is available showing that he's wrong. Your comment also made me recall another comment you wrote a couple of years ago about how my status in this community made a criticism of you feel like a "huge insult", which I couldn't understand at the time and just ignored.

I wonder how many other people feel this strongly about being criticized/insulted by a high status person (I guess at least Roko also felt strongly enough about being called "stupid" by Eliezer to contribute to him leaving this community a few days later), and whether Eliezer might not be aware of this effect he is having on others.

Here is clearer evidence that account deletion simply did nothing back then. My understanding is the same as komponisto's: Roko wrote a script to delete all of his posts/comments individually.

UDT shows how an agent might be able to care about something other than an externally provided reward, namely how a computation, or a set of computations, turn out. It's conjectured that arbitrary goals, such as "maximize the number of paperclips across this distribution of possible worlds" (and our actual goals, whatever they may turn out to be) can be translated into such preferences over computations and then programmed into an AI, which will then take actions that we'd consider reasonable in pursue of such goals.

(Note this is a simplification that ignores issues like preferences over uncomputable worlds, but hopefully gives you an idea what the "step" consists of.)

Would it be fair to say that your philosophy is similar to davidad's? Both of you seem to ultimately value some hard-to-define measure of complexity. He thinks the best way to maximize complexity is to develop technology, whereas you think the best way is to preserve evolution.

I think that evolution will lead to a local maximum of complexity, which we can't "help" it avoid. The reason is that the universe contains many environmental niches that are essentially duplicates of each other, leading to convergent evolution. For example Earth contains lots of species that are similar to each other, and within each species there's huge amounts of redundancy. Evolution creates complexity, but not remotely close to maximum complexity. Imagine if each individual plant/animal had a radically different design, which would be possible if they weren't constrained by "survival of the fittest".

Whereas the entire purpose of FAI is to trap the universe in a local maximum.

Huh? The purpose of FAI is to achieve the global maximum of whatever utility function we give it. If that utility function contains a term for "complexity", which seems plausible given people like you and davidad (and even I'd probably prefer greater complexity to less, all else being equal), then it ought to at least get somewhat close to the global complexity maximum (since the constraint of simultaneously trying to maximize other values doesn't seem too burdensome, unless there are people who actively disvalue complexity).