But since one common FAI proposal is "find out human preferences, and then do those", if it turns out human preferences don't really exist in a coherent way, that sounds like an important thing to know.

I would think that knowing evo psych is enough to realize this is a dodgy approach at best.

If you can opt out of it, it's not mandatory! You could get the best of both worlds, though: vitrify your head and donate the rest of your body. The only loss is, I think, your corneas.

If "humility" can be used to justify both activities and their opposites so easily, perhaps it's a useless concept and should be tabooed.

That's suggestion five on the list:

Make sure that each CSA above the lowest level actually has "could", "should", and "would" labels on the nodes in its problem space, and make sure that those labels, their values, and the problem space itself can be reduced to the managing of the CSAs on the level below.

Figuring out exactly how it is that our preferences, i.e., our utility function, emerge from the managing of our subagents is my main motivation for suggesting the construction of a parallel decision theory, as well as understanding how our problem space emerges from the managing of other CSAs.

Well, one story is that humans and brains are irrational, and then you don't need a utility function or any other specific description of how it works. Just figure out what's really there and model it.

The other story is that we're hoping to make a Friendly AI that might make rational decisions to help people get what they want in some sense. The only way I can see to do that is to model people as though they actually want something, which seems to imply having a utility function that says what they want more and what they want less. Yes, it's not true, people aren't that rational, but if a FAI or anyone else is going to help you get what you want, it has to model you as wanting something (and as making mistakes when you don't behave as though you want something).

So it comes down to this question: If I model you as using some parallel decision theory, and I want to help you get what you want, how do I extract "what you want" from the model without first somehow converting that model to one that has a utility function?

It's not pseudo profound, but, like The Matrix, it has a lot of window-dressing and pomous wanking around absolutely legitimate questions. It's also frustrating in that many of the questions are asked, but few are resolved. And they're mainly a framework for the character arcs to develop. Eva has a very simple plot, which it doses very carefully in order to make it more interesting, so that it comes off as a Jigsaw puzzle. The most interesting thing is how the characters evolve and... really, I don't want to spoil anything, but you should definitely give it a try: it's a character story where the characters are extremely human, layered, and rich, and their stories are extremely poignantes.

If you don't want to watch the original, all I can tell you is, the "inexplicable" turns out to be "not explained yet". Everything will be revealed in due time. As for why it is interesting... well, if you watch EVA, and especially the final movie, The End Of Evangelion, you might identify a lot with Shinji, put a lot of yourself into him. This is especially true if you watch it as a teenager of the same age. And then... well, stuff happens to him, and to you, by proxy. Seeing him well-adjusted, happy, strong, while still having the same fundamental character traits... it's a very intense feeling.

I find this whole line of conversation fairly ludicrous, but here goes:

Number 1. Time-inconsistency: we have different reactions about an immediate certainty of some bad than a future probability of it. So many people might be willing to go be a health worker in a poor country where aid workers are commonly (1 in 10,000) raped or killed, even though they would not be willing to be certainly attacked in exchange for 10,000 times the benefits to others. In the actual instant of being tortured anyone would break, but people do choose courses of action that carry risk (every action does, to some extent), so the latter is more meaningful for such hypotheticals.

Number 2. I have driven and flown thousands of kilometers in relation to existential risk, increasing my chance of untimely death in a car accident or plane crash, so obviously I am willing to take some increased probability of death. I think I would prefer a given chance of being tortured to a given chance of death, so obviously I care enough to take at least some tiny risk from what I said above. As I also said above, I'm not willing to make very big sacrifices (big probabilities of such nasty personal outcomes) for tiny shifts in probabilities of big impersonal payoffs (like existential risk reduction). In realistic scenarios, that's what "the cause" would refer to. I haven't made any verbal or explicit "precommitment" or promises or anything like that.

In sufficiently extreme (and ludicrously improbable) trolley-problem style examples, e.g. "if you push this button you'll be tortured for a week, but if you don't then the Earth will be destroyed (including all your loved ones) if this fair coin comes up heads, and you have incredibly (impossibly?) good evidence that this really is the setup" I hope I would push the button, but in a real world of profound uncertainty, limited evidence, limited personal power (I am not Barack Obama or Bill Gates), and cognitive biases, I don't expect that to ever happen. I also haven't made any promises or oaths about that.

I am willing to give of my time and effort, and forgo the financial rewards of a more lucrative career, in exchange for a chance for efficient do-gooding, interaction with interesting people who share my values, and a meaningful project. Given diminishing returns to money in rich countries today, and the ease of obtaining money for folk with high human capital, those aren't big sacrifices, if they are sacrifices at all.

Number 3. SIAIers love to be precise and analytical and consider philosophical thought experiments, including ethical ones. I think most have views pretty similar to mine, with somewhat varying margins. Certainly Michael Vassar, the head of the organization, is also keen on recognizing one's various motives and living a balanced life, and avoiding fanatics. Like me, he actively advocates Bostrom-like parliamentary model approaches to combining self-concern with parochial and universalist altruistic feelings.

I have never heard anyone making oaths or promises to make severe sacrifices.

Number 4. This is a pretty ridiculous question. I think that's fine and normal, and I feel more comfortable with such folk than the alternative. I think people should not exaggerate that do-gooding is the most important thing in their life lest they deceive themselves and others about their willingness to make such choices, which I criticized Roko for.

Hm, reading the original EVA is not compulsory, the story stands very well on its own... but since you ask, I heartily recommend you watch EVA and Gurren Lagann. They are both flawed, but they are still very good, and very memorable.

Your strength as a rationalist is your ability to be more confused by fiction than by reality.

Does that lead to the conclusion that Newcomb's problem is irrelevant? Mind-reading aliens are pretty clearly fiction. Anyone who says otherwise is much more likely to be schizophrenic than to have actual information about mind-reading aliens.

an inability to properly argue your point should have you questioning your point instead.

When dealing with trolls, whether on the Internet or in Real Life, no matter how absolutely damn sure you are of your point, you have no time to unravel their bullshit for what it is, and if you try it you will only bore your audience and exhaust their patience. Debates aren't battles of truth: there's publishing papers and articles for that. Debates are battles of status. If you manage to come off as the one with higher status, people will listen more to what you said during the debate, and, more importantly, to what you said afterwards.

A very interesting way of taking advantage of this and neutralizing the effects of the dirty fighting would be to immediately afterwards publish a play-by-play analysis of the discussion, using the opportunity as an occasion to teach those who were impressed by you and went to see your work how debate really works. You could even go so far as actually listing the arguments you and your opponents use, and openly admit it if your opponent's arguments are good enough that they have caused you to actually undertake a Bayesian update. That way, you show that:

*You're smart, witty, and charismatic enough to win the debate.

*You're rational, honest, and moral enough to admit to the truth afterwards.