The True Trolley Dilemma would be where the child is Eliezer Yudkowsky.

*Then* what would you do?

EDIT: Sorry if that sounds trollish, but I meant it as a serious question.

In response to
Readiness Heuristics

The True Trolley Dilemma would be where the child is Eliezer Yudkowsky.

*Then* what would you do?

EDIT: Sorry if that sounds trollish, but I meant it as a serious question.

I don't see any obvious reason why the answer to this question shouldn't be greater than the number of subatomic particles in your body.

Clarification: I am only talking about direct inputs to the decision making process, not what they're aggregated from (which would be the observable universe).

No, you can't ask yourself what you'll do. It's like a calculator that seeks the answers to the question of "what is 2+2?" in a form "what will I answer to the question "what is 2+2"?", in which case the answer 57 will be perfectly reasonable.

If you are cooperating with your copy, you only know that the copy will *do the same action*, which is a restriction on your joint state space. Given this restriction, the expected utility calculation for your actions will return a result different from what other restrictions may force. In this case, you are left only with 2 options: (C,C) and (D,D), of which (C,C) is better.

*reliably* win by determining whether the opponent one-boxes, we need to be Omega-superior relative to them, almost by the definition of Newcomb's. But such powers would allow us to just use the trivial solution: "cooperate if I think my opponent will cooperate".

Agreed that in general one will have some uncertainty over whether one's opponent is the type of algorithm who one boxes / cooperates / whom one wants to cooperate with, etc. It does look like you need to plug these uncertainties into your expected utility calculation, such that you decide to cooperate or defect based on your degree of uncertainty about your opponent.

However, in some cases at least, you don't need to be Omega-superior to predict whether another agent one-boxes....for example, if you're facing a clone of yourself; you can just ask yourself what you would do, and you know the answer. There may be some class of algorithms non-identical to you but which are still close enough to you to make this self-reflection increased evidence that your opponent will cooperate if you do.

In response to
Excuse me, would you like to take a survey?

Some little things:

- "Professional field" should be multiple-choice.
- What do you mean by "spiritual" under "religious views" – believe in the supernatural? take mysticism seriously in a way compatible with naturalism?
- On p(Aliens), does "the Universe" mean past light cone, present surface of observable universe, or entire (potentially infinite) continuum? How about other Everett branches?
- A definition of "supernatural" before the p(God) question would be nice.
- "Three Worlds Ending" might benefit from a "clear preference for specific other outcome" option.
- Similarly, at least some of the PD and other game theory/superrationality-related questions could have something like "different clear preferences depending on unspecified details of the situation".

In response to
Excuse me, would you like to take a survey?

In response to
Excuse me, would you like to take a survey?

Okay? Do whatever *you* want to do. If you know your expected value for your cryopreservation and and the expected value you have for the life-saving you could be doing with your organs then it's simple.

Eleizer's say so matters only in as much as he may be able to help with the math of translating your preferences into a coherent utility function.

To my knowledge, some of those issues remain unsolved, such as whether different simulations of oneself in different environments necessarily converge (seems to me very unlikely, and this looks provable in a simplified model of the situation), and if not, how to "best" harmonize their differing opinions...
similarly, whether a single simulated instance of oneself might itself not converge or not provably converge on one utility function as simulated time goes to infinity (seems quite likely; moreover, provable , in a simplified model) etc., etc.

If conclusive work has been done of which I'm unaware, it would be great if someone wants to link to it.

It seems unlikely to me that we can satisfactorily answer these questions without at least a detailed model of our own brains linked to reductionist explanations of what it means to "want" something, etc.

My point is slightly different from NFL theorems. They say if you exhaustively search a problem then there are problems for the way you search that mean you will find the optimum last.

I'm trying to say there are problems where exhaustive search is something you don't want to do. E.g. seeing what happens when you stick a knife into your heart or jumping into a bonfire. These problems also exist in real life, where as the NFL problems are harder to make the case that they exist in real life for any specific agent.

View more: Next

Two models can behave the same as what you've seen so far, but diverge in future predictions. Which model should you give greater weight to? That's the question I'm asking.

The current best answer we know seems to be to write each consistent hypothesis in a formal language, and weight longer explanations inverse exponentially, renormalizing such that your total probability sums to 1. Look up aixi, universal prior